Method and apparatus for transforming an initial image into a target image displayed by a display device

Abstract

The present invention provides improvements in and related to the display of images, and proposes a method for transforming an initial image into a target image for display by a display device, the method comprising: a first calculation for calculating a tone curve that maps the brightness levels of the original image to the brightness levels of a transformed image; and a first transformation for transforming the brightness levels of the original image according to the tone curve, thereby providing a first transformed image for display by the display device according to a peak brightness for display; the calculation includes determining the tone curve involving an optimization process, the optimization process optimizing the match between the contrast of the original image and the contrast of the first transformed image, wherein the peak brightness of the tone curve is less than or equal to the peak brightness of the first transformed image for display by the display device.

Description

[0001] This application is a divisional application of the invention patent application with application date of June 11, 2015, application number 201580042493.8, and invention name “Improvements in and related to image display”. Technical Field

[0002] The present invention relates to the display of image data on a display screen (eg via a display panel or by projection) and the processing of such image data. Background Art

[0003] The visual acuity of the eye can be measured by asking the subject to distinguish images of objects (e.g., letters or shapes placed on a white background). These tests are commonly employed in the evaluation of corrective lenses (e.g., glasses or contact lenses). Objects within an image can typically be better distinguished from the image background if they have a distinguishable brightness or color relative to the background. For example, relative brightness differences can be expressed in terms of a quantity known in the art as a "contrast ratio," or simply "contrast." It is typically defined in terms of the difference between two brightness values ​​divided by their sum.

[0004] Generally speaking, an object that is difficult to observe against its background will have little contrast. Experiments have shown that when the contrast of an object is less than a threshold (commonly referred to as the "contrast detection threshold" or "contrast threshold"), the eye cannot detect the object within the image. The inverse of this minimum perceptible contrast is commonly referred to as the eye's "contrast sensitivity."

[0005] In the past, test images containing test patterns have been used to investigate and quantify contrast sensitivity. These have generally included test patterns containing sinusoidal brightness variations that extend across the image in one dimension to form strips of continuously varying (rising and falling) brightness. For these brightness test patterns, contrast is simply defined as the amplitude of the sinusoid divided by the (uniform) mean of the sinusoid. The threshold amount of contrast required in this pattern for reliable detection / perception of contrast (e.g., sufficient to give a 50% probability of detection) is therefore called the contrast threshold. The contrast threshold of this test pattern depends on the wavelength of the sinusoidal variations in the image (i.e., the lateral spatial separation of the strips between consecutive brightness peaks). The inverse of this wavelength is called the "spatial frequency" of the pattern. Contrast sensitivity can also be measured using non-sinusoidal brightness variations, and in these cases, contrast can be defined as the difference between the maximum and minimum brightness in the image divided by their sum. This is called "Michelson contrast."

[0006] Models for various aspects of contrast sensitivity exist in the prior art for "photopic" brightness conditions (i.e., brightness conditions during daylight vision). These models are based on specific assumptions about the workings of the human eye. They provide mathematical expressions for quantifying the eye's contrast sensitivity. A key concept in these models is the assumption that contrast sensitivity is determined by noise in the visual system.

[0007] In practice, it has been found that there is no fixed contrast threshold below which a contrast pattern cannot be detected at all, but above which it can always be detected. Instead, there are gradually increasing probabilities of contrast detection. The contrast threshold is typically defined as the contrast at which a 50% probability of detection exists. Contrast values ​​below the contrast threshold will be detected with less than a 50% probability. The mathematical function that describes the probability of contrast detection as a function of contrast intensity is often called a "psychometric function." The statistical factors that determine the shape of the psychometric function are generally believed to be caused by noise that is partly internal to the visual system. One example of a psychometric function that has been successfully used in this context is the normal probability integral, a cumulative probability distribution function based on the well-known form of a Gaussian ("normal") probability density function centered on the contrast threshold. It is a function of the value of the image contrast in question, and it rises continuously as contrast increases, from a probability of 0.0 when the contrast is 0.0, through a value of 0.5 when the contrast equals the contrast threshold, to a value that asymptotically approaches 1.0.

[0008] Experiments suggest that, under photopic conditions, when the real / physical contrast values ​​(C) of the images in question do not actually reach their respective contrast thresholds (C T ), the apparent / perceived / visual contrast of the two sinusoidal patterns (Patterns 1 and 2) is perceived to be the same (i.e., matching) when the difference is ∫ ∫ . Thus:

[0009]

[0010] This means:

[0011]

[0012] Therefore, the perception caused by the physical contrast C is usually considered to be a function of its visual contrast (CC T ). The visual contrast in a sinusoidal image is at least assumed to always be reduced relative to its true / physical contrast by a contrast threshold and is proportional to the true / physical contrast of the image.

[0013] The brightness level in an image plays an important role in the perceived contrast of objects within that image. Images / scenes viewed under different brightness conditions are found to be perceived differently. The same physical scene viewed in bright sunlight and in dim conditions does not appear the same to the human eye. Similarly, images shown on a bright image display and on a relatively dimly lit cinema screen differ significantly in their appearance.

[0014] Color and contrast perception vary significantly across a range of illumination levels. When luminance drops to 3-5 cd / m², the cones of the retina steadily lose their sensitivity and the visual signal is influenced by the rods of the retina. 2 Below 100 nm, the most dramatic changes in vision are observed. Here, a gradual loss of the so-called "mesopic" visual range, acuity, and color vision occurs. This important characteristic of the visual system is rarely considered when reproducing colors on electronic displays. Although state-of-the-art display colorimetry is almost entirely based on cone-mediated vision (CIE color matching functions), a significant portion of the color gamut in modern displays is generally located within the 3 cd / m² range, which is partially mediated by rods. 2 This is particularly relevant for mobile phone displays, which may reduce their brightness by 10-30 cd / m² below peak luminance. 2 This means that in the case of dimmed high-contrast displays, approximately ¾ of the perceived color gamut cannot be accurately reproduced using conventional cone-based colorimetry.

[0015] The present invention aims to address these limitations of the prior art relating particularly, but not exclusively, to mesopic vision. Summary of the Invention

[0016] In one aspect, a brightness recalibration method is implemented below for altering the perceived contrast and / or color of images to match their appearance at different brightness levels. The invention preferably employs a psychophysical method of matching contrast. The method may take into account the rod contribution to vision (photoreceptors). The recalibration preferably comprises finding an optimized tone curve, and / or preferably spatial contrast manipulation, and / or preferably adjusting the color hue and / or color saturation in the image to be displayed. This allows for adjusting or providing an image that reliably simulates night vision under bright conditions, or compensating for a bright image displayed on a darker display so that it reveals details and / or colors that would otherwise be invisible.

[0017] In a second aspect, a method for locally transforming an image within a sub-block of an image to adjust the contrast of the image for display by a display device may be provided below, comprising: calculating a contrast adjustment factor for adjusting the contrast within a sub-block of an original image; and transforming the contrast within the sub-block of the original image according to the contrast adjustment factor, thereby providing a transformed image for display by the display device; wherein the calculating comprises determining a measure of local contrast within the sub-block, and determining the contrast adjustment factor accordingly to optimize the match between the contrast of the original image and the contrast of the transformed image within the sub-block.

[0018] In a third aspect, a method for transforming an image of a first brightness to adjust its perceived color hue for display by a display device according to a second brightness may be provided below, the method comprising: calculating a color adjustment factor for adjusting the color values ​​of the original image; and adjusting the color values ​​of the original image according to the color adjustment factor to thereby provide a transformed image for display by the display device according to the second brightness; and wherein the calculating comprises: numerically representing the cone photoreceptor response to the color value in view of the corresponding contributing rod photoreceptor response to the brightness.

[0019] In a fourth aspect, a method for transforming an image having a first brightness to adjust its color saturation for display on a display device having a second brightness may be provided below, the method comprising: calculating a color saturation adjustment transform for adjusting color values ​​of an original image; and adjusting the color values ​​of the original image according to the color saturation transform. Thus, a transformed image is provided for displaying the display device at the second brightness; wherein, according to the following transformation, the value of the first brightness (Y) and the second brightness are transformed. The adjusted color value is defined by the value of and the saturation correction factor (s(…)):

[0020]

[0021] The saturation correction factor is a function of brightness and approaches a value of zero as brightness approaches zero, and asymptotically and monotonically approaches a value of 1 (1.0) as brightness increases. This unusual form of approaching zero as a function of decreasing brightness has been discovered experimentally and has proven to be surprisingly effective in correcting color saturation.

[0022] In a fifth aspect, the following may provide an apparatus for transforming an image for display by a display device according to a peak brightness for display, the apparatus comprising: a calculation unit for calculating a tone curve for mapping the brightness level of an original image to the brightness level of a transformed image; and a transformation unit for transforming the brightness level of the original image according to the tone curve, thereby providing a transformed image for display by the display device; wherein the calculation unit is arranged to determine a tone curve that optimizes the match between the contrast of the original image and the contrast of the transformed image.

[0023] In a sixth aspect, a device for locally transforming an image within a sub-block of an image for display by a display device to adjust the image contrast may be provided below, comprising: a calculation unit for calculating a contrast adjustment factor for adjusting the contrast within a sub-block of the original image; and a transformation unit for transforming the contrast within the sub-block of the original image according to the contrast adjustment factor, thereby providing a transformed image for display by the display device; wherein the calculation unit is arranged to: determine a measure of the local contrast within the sub-block, and based on this, determine a contrast adjustment factor that optimizes the match between the contrast of the original image and the contrast of the transformed image within the sub-block.

[0024] In a seventh aspect, a device for transforming an image of a first brightness to adjust its perceived color hue for display by a display device according to a second brightness may be provided below, the device comprising: a calculation unit for calculating a color adjustment factor for adjusting the color values ​​of the original image; and an adjuster unit for adjusting the color values ​​of the original image according to the color adjustment factor, thereby providing a transformed image for display by the display device according to the second brightness; wherein the calculation unit is arranged to numerically represent the cone photoreceptor response to the color value in view of the corresponding contribution of the rod photoreceptor response to the brightness.

[0025] In its eighth aspect, there may be provided an apparatus for transforming an image having a first brightness to adjust its color saturation for display on a display device having a second brightness, the method comprising: a calculation unit for calculating a color saturation adjustment transform for adjusting color values ​​of an original image; and an adjuster unit for adjusting the color values ​​of the original image according to the color saturation transform. Thus, a transformed image is provided for displaying the display device at the second brightness; wherein the adjuster unit is arranged to: transform the image according to the value of the first brightness (Y) and the second brightness according to the following The color value is adjusted by the value of and the saturation correction factor (s(…)):

[0026]

[0027] The saturation correction factor is a function of brightness, approaches a value of zero as the brightness approaches zero, and gradually and monotonically approaches a value of 1 (1.0) as the brightness increases.

[0028] In another aspect, an apparatus for performing the above method may be provided below.

[0029] In yet another aspect, the following may provide a computer program or computer program product comprising computer executable instructions arranged to implement the method as described above when executed in a computer. The present invention may provide a computer programmed to implement the method as described above.

[0030] In another aspect, a method for adjusting data for an image displayed by a display device according to external lighting conditions may be provided below, the method comprising: providing first brightness data representing a first brightness level of pixels of an image suitable for display under a first external lighting; providing second brightness data representing a brightness level of pixels of the image different from the first brightness data and suitable for display under a second external lighting different from the first external lighting; adjusting the brightness level of the first brightness data so that the image contrast within the entire image represented by the adjusted first brightness data substantially matches the corresponding image contrast within the entire image represented by the second brightness data; determining the background brightness within the entire image represented by the adjusted first brightness data; defining an image sub-region within the image, and adjusting the brightness level of the first brightness data associated with the image sub-region so that the image contrast locally to the image sub-region substantially matches the corresponding image contrast locally to the image sub-region represented by the second brightness data of the image; and using the background brightness of the image sub-region and the adjusted first brightness data to generate brightness image data for use in displaying the image under the second external lighting.

[0031] In another aspect, a method for transforming an image for display by a display device according to a peak brightness for display may be provided below, the method comprising: calculating a tone curve that maps the brightness levels of an original image to the brightness levels of a transformed image; and transforming the brightness levels of the original image according to the tone curve, thereby providing the transformed image for display by the display device; wherein the calculation comprises: determining a tone curve that optimizes the match between the observer sensitivity and / or adaptability of the brightness levels for the original image to contrast and the observer sensitivity and / or adaptability of the brightness levels for the transformed image to contrast.

[0032] In yet another aspect, a method for locally transforming an image within a sub-block of an image for display by a display device to adjust image contrast may be provided, comprising: calculating a contrast adjustment factor for adjusting the contrast within a sub-block of an original image; and transforming the contrast within the sub-block of the original image according to the contrast adjustment factor to thereby provide a transformed image for display by the display device; wherein the calculating comprises determining a measure of local contrast within the sub-block, and determining therefrom a contrast adjustment factor that optimizes a match between observer sensitivity and / or adaptability of brightness levels for contrast for the original image and observer sensitivity and / or adaptability of brightness levels for contrast for the transformed image within the sub-block.

[0033] In another aspect, the present invention may provide a method for transforming an initial image into a target image for display by a display device, the method comprising: a first calculation for calculating a tone curve that maps the brightness level of the original image to the brightness level of a transformed image; and a first transformation for transforming the brightness level of the original image according to the tone curve, thereby providing a first transformed image for display by the display device according to a peak brightness for display; the calculation comprising determining the tone curve involving an optimization process, the optimization process optimizing the match between the contrast of the original image and the contrast of the first transformed image, wherein the peak brightness of the tone curve is less than or equal to the peak brightness of the first transformed image for display by the display device.

[0034] In another aspect, the present invention may provide a display device for transforming an initial image into a target image for display, wherein the initial image has a contrast value and / or a color value, and the display device includes a calculation unit and a transformation unit, the calculation unit includes a global contrast recalibration unit, and the global contrast recalibration unit is arranged to: receive the brightness (Y) of the initial image to be transformed and the target brightness to which the initial image to be transformed is related as input, so that the contrast value and / or color value of the initial image to be transformed to present the target image better for viewing at the target brightness level. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Shown are the original image (center) and two recalibrated images (left, right);

[0036] Figure 2 A schematic flow chart of a recalibration method according to a preferred embodiment of the present invention is shown;

[0037] Figure 3 Two contrast sensitivity functions (CFS) are shown as a function of image brightness (left) and image spatial frequency (right);

[0038] Figure 4 showing a graph illustrating lines of matching contrast magnitude as a function of brightness;

[0039] Figure 5 Two patch linear tone curves for an image are shown;

[0040] Figure 6 shows contrast matching data according to four different contrast matching methods;

[0041] Figure 7 FIG. 1 shows a tone curve adjusted according to a preferred embodiment of the present invention for recalibrating the brightness of an image;

[0042] Figure 8 shows an adjusted image resulting from an implementation of the method according to a preferred embodiment of the invention;

[0043] Figure 9 shows the effect of contrast rescaling of image edge features according to various methods;

[0044] Figure 10 shows the spectral emissions of three different image display panels;

[0045] Figure 11 showing a curve representing a change in color saturation correction according to a change in brightness;

[0046] Figure 12Shown are images compensated for viewing by a younger (left) viewer and an older (right) viewer;

[0047] Figure 13 A comparison of an original image (left), the image adjusted according to existing methods (center), and the image adjusted according to the present invention (right) is shown;

[0048] Figure 14 shows a comparison of an original image (left), the image adjusted according to the present invention to represent nighttime viewing (center), and an image adjusted according to the present invention to represent dramatic / exaggerated viewing (right);

[0049] Figure 15 A comparison of an original image (left), the image adjusted according to existing methods (center), and the image adjusted according to the present invention (right) is shown;

[0050] Figure 16 shows a comparison of images adjusted according to the existing method (first 5 rows) and images adjusted according to the present invention (bottom row);

[0051] Figure 17 Shown is a comparison of experimental data associated with rescaling a bright image with respect to a dark display (left) and a comparison of experimental data associated with rescaling a dark image with respect to a bright display (right). DETAILED DESCRIPTION

[0052] As described above, in one aspect, a brightness recalibration method is implemented below for altering the perceived contrast and / or color of images to match their appearance at different brightness levels. The invention preferably employs a psychophysical method of matching contrast. In a preferred embodiment, the invention may take into account the rod contribution to vision (photoreceptors). The recalibration preferably includes finding an optimized tone curve, and / or preferably spatial contrast processing, and / or preferably adjusting the color hue and / or color saturation in the image to be displayed. This allows for adjusting or providing an image that reliably simulates night vision under bright conditions, or compensating for a bright image displayed on a darker display so that it reveals details and / or colors that would otherwise be invisible.

[0053] To account for changes in image appearance due to lower absolute brightness levels, the present invention preferably implements a novel appearance matching method and brightness recalibration method. The method can be used to compensate for changes in appearance between brightness levels, thus allowing for further reductions in display brightness and thus power savings. The method can also be used to perform recalibration in the opposite direction, from a dark scene to a much brighter display, to reproduce the appearance of a night scene. The method preferably accounts for color and contrast perception across the entire brightness range. Changes in overall brightness and contrast can preferably be compensated for by optimizing the shape of the tone curve, thereby providing a better compromise between maintaining contrast and brightness while utilizing the available dynamic range of the display. Changes in detail visibility can be accounted for using a novel visual contrast matching method. Changes in color appearance can preferably be accounted for by representing the rod contribution and loss of color saturation at low brightness levels. Each of these components, individually or in combination, preferably provides superior appearance matching across the brightness range that is not achievable by existing methods.

[0054] In a first aspect, the present invention may provide a method for transforming an image for display by a display device according to a peak brightness for display, the method comprising: calculating a tone curve that maps the brightness level of an original image to the brightness level of a transformed image; and transforming the brightness level of the original image according to the tone curve to provide a transformed image for display by the display device; wherein the calculation comprises: determining a tone curve that optimizes the match between the contrast of the original image and the contrast of the transformed image, wherein the peak brightness of the tone curve does not exceed the peak brightness of the transformed image for display by the display device.

[0055] In this way, a suitably constrained tone curve can be used to form a mechanism for calculating and applying contrast adjustments to an image. Calculation of the tone curve can be performed on a piecewise basis, where the tone curve is represented as a piecewise linear function having a plurality of linear segments each representing a particular tonal range, and where the determination of the linear segments is performed according to the above optimization process. Once all linear segments of the tone curve are calculated, a complete tone curve covering all relevant tones is also calculated. If desired, the tone curve can be efficiently / practically rendered nonlinear by making the linear segments small enough in width and large enough in number to approximate a continuous curve, however, this can be more computationally intensive.

[0056] The contrast of the original image is preferably a visual contrast value as the difference between the "physical" contrast value for the original image and the detection threshold contrast value for the original image. For example, the two contrasts matched according to the above optimization can be defined as the general formula:

[0057] Visual contrast = CC T

[0058] Among them, C is the physical contrast value as a property of the image, C T is the contrast detection threshold value. It has been found that applying optimization techniques with respect to visual contrast is most effective. Accordingly, the contrast of the transformed image is preferably also a visual contrast value that is the difference between the physical contrast value for the transformed image and the detection threshold contrast value for the original image. Therefore, the optimized matching preferably includes minimizing the difference between the visual contrast of the original image and the visual contrast of the transformed image. The goal of the optimization is generally to achieve the closest suitable approximation to the following conditions:

[0059]

[0060] or,

[0061]

[0062] Regarding images 1 (original) and 2 (transformed).

[0063] The optimized match may include minimizing the sum of the squared differences between the visual contrast of the original image and the visual contrast of the transformed image for a plurality of different luminance values ​​within the original image. For example, because a perfect match between the contrasts of the positive matches may not be achieved for all luminance levels associated with the calculated tone curve, it may be found that the closest numerical approximation to a perfect match (Δ=0) may be a value of Δ that fluctuates between small positive and small negative values ​​over the range of luminances considered with respect to the tone curve. By summing the squared values ​​of all these fluctuations (ΣΔ 2 ) is optimized to be as small as possible, we can effectively optimize jointly with respect to the tone curve traversing the brightness range.

[0064] The optimization can be implemented as the minimization / optimization of the following formula:

[0065]

[0066] or:

[0067]

[0068] or:

[0069]

[0070] It may preferably be subject to:

[0071]

[0072] and preferably also subject to:

[0073] T(l min )≥d min , T(l max )≤d max

[0074] Among them, for the logarithmic brightness l, G t (l) is the threshold contrast, and T(l) is the tone curve to be determined. The term τ is a constant that can be between 0.001 and 0.00001 (e.g., approximately 0.0001). In addition, l min and l max is the minimum and maximum brightness in the original image, d min and d max are the minimum and maximum values ​​for brightness within the transformed image for display by the display device (eg, limits set by the display device).

[0075] The detection threshold contrast value for the original image and / or the transformed image is preferably defined by a predetermined contrast sensitivity function as a function of brightness and spatial frequency. t It can be defined as:

[0076]

[0077] Here, "S" is an absolute sensitivity factor that is preferred for adjusting the absolute threshold value for a specific experimental situation. The value of S in a preferred embodiment can be between 8.0 and 9.0, and most preferably, S=8.6 or thereabouts. The threshold contrast value can be defined in logarithmic luminance space via the following formula:

[0078] Among them, contrast is usually

[0079] There are many contrast sensitivity functions (CSFs) known in the art and which can be appropriately selected by a person skilled in the art, however, a CSF of the following form has been found to be effective:

[0080]

[0081] The modulation transfer function (MTF), which models the visual effects / influence on contrast sensitivity caused by the scattering of light produced by / within the eye, can be defined as a function of the image spatial frequency (ρ) as follows:

[0082]

[0083] And the combined brightness sensitivity curve for cone and rod photoreceptors can be given by;

[0084]

[0085] The variable ρ is the spatial frequency (cycles per degree), and l is the logarithmic brightness (l = log 10 (Y)), where Y is the brightness value. Parameters p1 to p8 and a1 to a4 are fitting parameters that can be selected by the user according to preference. However, an example of this CSF is shown here. Figure 3 and are discussed in detail below:

[0086] MANTIUK,R.,KIM,KJ,REMPEL,AG,AND HEIDRICH,W.2011.HDR-VDP-2:Acalibrated visual metric for visibility and quality predictions in allluminance conditions.ACM Trans.Graph(Proc.SIGGRAPH)30,4(July2011),1.

[0087] Examples of numerical values ​​for these parameters are as follows.

[0088] MTF - Modulation Transfer Function of the Eye

[0089]

[0090]

[0091] CSF - (neural) contrast sensitivity function

[0092]

[0093] The values ​​of the parameters p1 to p4 used in the CSF are given above and are the luminance (L a ) function. In order to obtain the function for p with respect to a given brightness 1- For the values ​​of p4, we can interpolate between the tabulated values ​​(in log-luminance space).

[0094] s A - Combined brightness sensitivity

[0095]

[0096] An alternative to CSF ​​is Barten CSF from:

[0097] BARTEN,PGJ(1999).Contrast sensitivity of the human eye and its effects on image quality(p.208).SPIE Press.

[0098] The calculation preferably includes: transforming the brightness level (Y) of both the original image and the transformed image into a value according to l=log 10 (Y) and calculate the tone curve with respect to the logarithmic brightness level. This operation has advantages in terms of numerical value. For example, the model expressed by the logarithmic contrast value does not experience singularities at high contrast values.

[0099] Furthermore, the slope of the tone curve in the logarithmic domain corresponds to a change in contrast. This property can be exploited to calculate the contrast of the transformed image as the product of the contrast of the original image and the slope of the tone curve for a given brightness. This simplifies the process of calculating the optimized tone curve.

[0100] The present invention, in its first aspect, may therefore provide a way of globally adjusting the contrast of an image to optimize the visual contrast of the image based on the maximum brightness of the hue of the adjusted image to be displayed by the display device (the maximum value of the tone curve). This operation may be particularly useful when modifying images that were initially intended / prepared for viewing on a bright image display / screen so that they can be optimally viewed on a dimmed display (for example, more suitable in a low light environment). A well-chosen tone curve can greatly improve the appearance of the modified / transformed image. However, in a second aspect, the present invention may provide a method of transforming local features of an image that takes into account the contrast levels and spatial frequencies of the image separately in local areas. This local transformation technique may be applied alone or in combination with the image transformation technique of the invention in its first aspect - that is, as a combined process for transforming an image both globally and locally.

[0101] In a second aspect, the present invention may provide a method for locally transforming an image within a sub-block of an image to adjust the contrast of the image for display by a display device, comprising: calculating a contrast adjustment factor for adjusting the contrast within a sub-block of an original image; and transforming the contrast within the sub-block of the original image according to the contrast adjustment factor to thereby provide a transformed image for display by the display device; wherein the calculation comprises: determining a measure of local contrast within the sub-block, and determining therefrom a contrast adjustment factor for optimizing the match between the contrast of the original image and the contrast of the transformed image within the sub-block, wherein the brightness in the sub-block of the original image does not match the brightness in the sub-block of the transformed image for display by the display device.

[0102] In this way, a local contrast matching method is provided in which local contrast can be measured. Measuring local contrast can be in terms of measuring the local variation (e.g., variance or standard deviation) of pixel brightness levels within a defined local block, and this can be used to optimize the matching of contrast between the original image and the transformed image within the local block. For example, a matching optimization can be performed on pixel-by-pixel levels, wherein for pixels within a given local block, a measure of local contrast is determined in terms of the local variation in brightness within the block associated with the pixel. The measure of local contrast can be any suitable measure of contrast or brightness variation that will be readily understood by those skilled in the art (e.g., using existing definitions of contrast, or in terms of variance or standard deviation of brightness values, or using values ​​provided by decomposition into Laplace (Difference of Gaussian) cones within a defined local block).

[0103] Preferably, the contrast of the original image is also a visual contrast value as the difference between a physical contrast value for the original image and a detection threshold contrast value for the original image. Preferably, the contrast of the transformed image is also a visual contrast value as the difference between a physical contrast value for the transformed image and a detection threshold contrast value for the transformed image. Therefore, the optimized matching preferably includes minimizing the difference between the visual contrast of the original image and the visual contrast of the transformed image. The contrast adjustment factor is preferably determined so that the difference between the value of the measure of local contrast (e.g., c) and the value of the product (e.g., c×m) of the measure of local contrast and the adjustment factor (e.g., m) substantially matches the detection threshold contrast value (e.g., ) and the detection threshold contrast value used for the original image (e.g. G t ) between .

[0104] The detection threshold contrast value for the original image and / or the transformed image is preferably defined by a predetermined contrast sensitivity function as a function of brightness and spatial frequency. There are many contrast sensitivity functions (CSFs) known in the art and which can be appropriately selected by a person skilled in the art, however, it has been found that a CSF of the following form is effective:

[0105]

[0106] The modulation transfer function (MTF), which models the visual effects / influence on contrast sensitivity caused by the scattering of light produced by / within the eye, can be defined as a function of the image spatial frequency (ρ) as follows:

[0107]

[0108] And the combined brightness sensitivity curve for cone and rod photoreceptors can be given by;

[0109]

[0110] The variable ρ is the spatial frequency (cycles per degree), and l is the logarithmic brightness (l = log 10 (Y)), where Y is the brightness value. Parameters p1 to p8 and a1 to a4 are fitting parameters that can be selected by the user according to preference.

[0111] Detection threshold contrast value M t It can be defined as:

[0112]

[0113] Here, "S" is an absolute sensitivity factor that is preferred for adjusting the absolute threshold value for a specific experimental situation. The value of S in a preferred embodiment can be between 8.0 and 9.0, and most preferably, S=8.6 or thereabouts. The threshold contrast value can be defined in logarithmic luminance space via the following formula:

[0114] Among them, contrast is usually

[0115] The values ​​of the parameters p1 to p4 used in the CSF are given above and are the luminance (L a To get values ​​for p1-p4 for a given luminance, we can interpolate between the tabulated values ​​(in log luminance space).

[0116] The sub-blocks are preferably defined by a spatial window function centered thereon, wherein the width of the window function is proportional to the inverse of the spatial frequency of the original image, such that the width of the window function is smaller for higher spatial frequencies.

[0117] The spatial window function can be any suitable window function, such as one skilled in the art will readily appreciate. The spatial window function can be zeroed outside the defined sub-block. For example, it can be bell-shaped, rectangular, or triangular, or other shapes. Examples include a two-dimensional Gaussian window function, a Parzen window function, or a Hamming window function, or other spatial window functions. For example, a two-dimensional Gaussian window function g σ It can be applied to a spatial function f(x,y) (e.g., a luminance image: f(x,y)=l(x,y)) in the following way:

[0118] g σ *f(x,y)=∫∫f(x-u1,y-u2)G σ (u1)G σ (u2)du1du2

[0119] Wherein, for i=1, 2, the Gaussian function defining the window and having a standard deviation σ is expressed as:

[0120]

[0121] The function G may be another window function other than Gaussian. A measure of local contrast (c) may be determined from the brightness (l) of the pixel values ​​of the spatial window function (g) defining the sub-block and centered on (x, y) according to the following formula:

[0122]

[0123] Wherein the operator (*) is a convolution operator. The spatial window function can be a Gaussian kernel with a standard deviation σ. The Gaussian window can be controlled / adjusted to become smaller with respect to higher spatial frequencies to account for finer scales. This can be achieved by making it equal to half the size of a single cycle at a particular spatial frequency:

[0124]

[0125] Among them, R ppd is the angular display resolution of the display used to display the image in pixels per visual degree, and p is the spatial frequency in cycles per degree.

[0126] The calculation preferably includes: transforming the brightness level (Y) of both the original image and the transformed image into a value according to l=log 10(Y) is a logarithmic luminance value (l) defined by the method, and the adjustment factor is calculated with respect to the logarithmic luminance value.

[0127] The image to be transformed may be decomposed into a plurality of component images, each of which corresponds to the original image described above and may be processed separately therefrom. Accordingly, the calculation preferably comprises decomposing the original image into an image cone comprising a plurality of different component images each representing the original image via spatial frequencies within respective ones of a plurality of different spatial frequency bands. Calculating the adjustment factors may comprise calculating respective adjustment factors with respect to one or each component image. The transformation may comprise transforming one or each component image, and may further comprise recombining the transformed image from the plurality of transformed component images. For example, the method may comprise decomposing the image into a Difference of Gaussian (Laplacian) image cone, and then manipulating the pixel values ​​of each image cone level so that the visual contrast of a given original cone level image and a corresponding transformed cone level image substantially matches, and then recombining / reconstructing the transformed image using the transformed cone level images.

[0128] The contrast adjustment factor can be implemented as:

[0129]

[0130] Among them, c k (x,y) is the contrast at pixel position (x,y) and level k of the image pyramid, where k = 1...N. The value of N may be chosen such that the coarsest band (optionally excluding the baseband) has a peak frequency less than or equal to 2 cpd.

[0131] Contrast adjustment as a local enhancement of an image (e.g., Laplacian) cone can be implemented as:

[0132]

[0133] Among them, P k Of course, in other embodiments, the image pyramid is not used. In this case, the above expression can be realized by setting N=1, and P k＝1 Corresponds to the original image level (without cones).

[0134] The method may further comprise replacing the component image associated with the lowest spatial frequency with a background luminance (e.g., a baseband image) derived from the original image transformed according to the first aspect of the invention. For example, the baseband of the transformed image according to the first aspect of the invention may be used to reconstruct an image using the transformed pyramid level image. The calculation preferably comprises expressing the contrast of the transformed image or the transformed component image in terms of (e.g., as) the product of the contrast of the original image and the value of the adjustment factor.

[0135] In a third aspect, the present invention may provide a method for transforming an image at a first brightness to adjust its perceived color hue for display by a display device at a second brightness, the method comprising: calculating a color adjustment factor for adjusting color values ​​of an original image; and adjusting the color values ​​of the original image according to the color adjustment factor to thereby provide a transformed image for display by the display device at the second brightness; and wherein the calculating comprises numerically representing cone photoreceptor responses to color values ​​given corresponding contributing rod photoreceptor responses to brightness, and wherein the cone photoreceptor response per unit brightness at the second brightness is constrained to substantially match the cone photoreceptor response per unit brightness at the first brightness. The contributing rod photoreceptor response is preferably a brightness-dependent value added to the cone photoreceptor response.

[0136] The step of numerically representing the cone photoreceptor responses preferably comprises separately representing the individual responses of the L cone, M cone and S cone, each in view of the respective corresponding contributions to luminance of the rod photoreceptor responses. Preferably, the color values ​​are primary color values ​​(eg RGB color values).

[0137] In this way, for example, color adjustment can include: converting the original image into cone responses and rod responses, and then calculating the rod contributions to the long wavelength (visible light), medium wavelength (visible light), and short wavelength (visible light) cone responses depending on the brightness of the original image, and adding the rod contributions to the long wavelength, medium wavelength, and short wavelength cone responses.

[0138] The rod photoreceptor response E can be expressed as R Each cone channel (E L 、E M 、E S ) of the photoreceptor responses (L, M, S).

[0139] For example:

[0140] L=E L +k0E R

[0141] M=EM +k1E R

[0142] S=E S +k2E R

[0143] Here, k i (i=0, 1, 2) is a weighting factor. Preferably, the weighting factor is dependent on the brightness (Y). Preferably, k0=k1. Preferably, k2 is different from k0 and k1. For example, when k0=k1, the value of the weighting factor is dependent on the brightness as follows:

[0144] <![CDATA[Y[cd / m 2 ]]]> 10 0.62 0.10 <![CDATA[k1]]> 0 0.0173 0.173 <![CDATA[k2]]> 0 0.0101 0.357

[0145] The responses to the original image and the transformed image are preferably normalized by the brightness of the transformed image or the original image, respectively, and the normalized response to the transformed image is adjusted to match the normalized response to the original. The resulting matched normalized transformed image response is then converted back to RGB values.

[0146] In a fourth aspect, the present invention may provide a method for transforming an image having a first brightness to adjust its color saturation for display on a display device having a second brightness, the method comprising: calculating a color saturation adjustment transform for adjusting color values ​​of an original image; and adjusting the color values ​​of the original image according to the color saturation transform. Thus, a transformed image is provided for display by the display device at the second brightness; wherein the adjusted color value is defined according to the value of the first brightness (Y) and the value of the second brightness (Y) and the saturation correction factor (s(...)) according to the following transformation:

[0147]

[0148] The saturation correction factor is a function of brightness and approaches a value of zero as brightness approaches zero, and asymptotically and monotonically approaches a value of 1 (1.0) as brightness increases. This unusual form of approaching zero as a function of decreasing brightness has been discovered experimentally and has proven to be surprisingly effective in correcting color saturation.

[0149] Color Value Preferably, it is a three-primary color value (eg RGB).

[0150] The method may include: determining an average brightness of an original image having a first brightness and determining an average brightness of an image having the first brightness, determining respective values ​​of a saturation correction factor (s(…)) according to each of the average brightnesses, and adjusting a color value using the respective values ​​of the saturation correction factor.

[0151] The original image may be an image adjusted or transformed according to the method of the present invention in its first aspect or second aspect. Thus, the present invention may provide a color saturation adjustment method, comprising: determining a color saturation correction for the original image; determining a color saturation correction for the contrast-transformed image according to the first aspect or second aspect of the present invention; and applying the color saturation correction to pixel color values ​​based on a ratio of the saturation correction attributed to the brightness of the original image to the saturation correction attributed to the brightness of the contrast-transformed image.

[0152] The original image may be an image adjusted or transformed according to the method of the present invention in its third aspect. Thus, color saturation correction may be applied to the color hue corrected original image.

[0153] The above color processing can not only improve color matching but also reduce the blue cast of images when viewed in the dark. This is desirable for two reasons. First, it puts less stress on the rods, which are very sensitive to blue. Second, these images are less likely to interfere with the light-sensitive retinal ganglion cells that are responsible for our brain's circadian clock. Some cases of insomnia are attributed to the high levels of near-blue light from TVs and mobile devices that people use at night.

[0154] These are two added benefits of the present invention in its related aspects.

[0155] In a fifth aspect, the present invention may provide an apparatus for transforming an image for display by a display device according to a peak brightness for display, the apparatus comprising: a calculation unit for calculating a tone curve for mapping the brightness level of an original image to the brightness level of a transformed image; and a transformation unit for transforming the brightness level of the original image according to the tone curve, thereby providing a transformed image for display by the display device; wherein the calculation unit is arranged to: determine a tone curve that optimizes the match between the contrast of the original image and the contrast of the transformed image, wherein the peak brightness of the tone curve does not exceed the peak brightness of the transformed image for display by the display device.

[0156] The contrast of the original image is preferably a visual contrast value which is the difference between a physical contrast value for the original image and a detection threshold contrast value for the original image.

[0157] The contrast of a transformed image is a visual contrast value that is the difference between a physical contrast value for the transformed image and a detection threshold contrast value for the transformed image.

[0158] The calculation unit is preferably arranged to perform the step of optimizing the match by a process comprising minimizing the difference between the visual contrast of the original image and the visual contrast of the transformed image.

[0159] The computation unit may be arranged to perform the step of performing the optimized matching by a process comprising minimizing the sum of squared differences between the visual contrast of the original image and the visual contrast of the transformed image with respect to a plurality of different luminance values ​​within the original image.

[0160] The detection threshold contrast value for the original image and / or the transformed image is preferably defined by a predetermined contrast sensitivity function as a function of brightness.

[0161] The calculation unit is preferably arranged to transform the brightness level (Y) of both the original image and the transformed image into a value according to l=log 10 (Y) defines the logarithmic luminance value (l), and calculates the tone curve with respect to the logarithmic luminance value.

[0162] The calculation unit may be arranged to calculate the contrast of the transformed image as a product of the contrast of the original image and a value of the slope of the tone curve with respect to a given brightness.

[0163] In a sixth aspect, the present invention may provide an apparatus for locally transforming an image within a sub-block of an image for display by a display device to adjust the image contrast, comprising: a calculation unit for calculating a contrast adjustment factor for adjusting the contrast within a sub-block of an original image; and a transformation unit for transforming the contrast within the sub-block of the original image according to the contrast adjustment factor, thereby providing a transformed image for display by the display device; wherein the calculation unit is arranged to: determine a measure of local contrast within the sub-block, and based on this, determine a contrast adjustment factor that optimizes the match between the contrast of the original image and the contrast of the transformed image within the sub-block, wherein the brightness in the sub-block of the original image does not match the brightness in the sub-block of the transformed image for display by the display device.

[0164] The calculation unit is preferably arranged to determine the contrast adjustment factor such that a difference between a value of the measure of local contrast and a value of a product of the measure of local contrast and the adjustment factor substantially matches a difference between a detection threshold contrast value for the transformed image and a detection threshold contrast value for the original image.

[0165] The calculation unit is preferably arranged to define the subblocks by a spatial window function centered on (x, y), wherein the width of the window function is proportional to the inverse of the spatial frequency of the original image, such that the width of the window function is smaller for higher spatial frequencies.

[0166] The calculation unit may be arranged to determine a measure of local contrast (c) with respect to the brightness (l) of the pixel values ​​from a spatial window function (g) defining the subblock and centered on (x, y) according to the following formula:

[0167]

[0168] Here, the operator (*) is the convolution operator.

[0169] The calculation unit may be arranged to define a detection threshold contrast value for the original image and / or said transformed image by a predetermined contrast sensitivity function as a function of brightness and spatial frequency.

[0170] The calculation unit is preferably arranged to transform the brightness level (Y) of both the original image and the transformed image into a value according to l=log 10 (Y) is a logarithmic luminance value (l) defined by the luminance_value_s, and an adjustment factor is calculated with respect to the logarithmic luminance value.

[0171] The image to be transformed may be decomposed into a plurality of component images, each of which corresponds to the original image described above and may be processed individually accordingly. Accordingly, the calculation unit is preferably arranged to decompose the original image into an image cone comprising a plurality of different component images each representing the original image via spatial frequencies within respective ones of a plurality of different spatial frequency bands. The calculation unit is preferably arranged to calculate the adjustment factors by a process comprising calculating respective adjustment factors for the or each component image. The transformation unit is preferably arranged to transform the or each component image and may further be arranged to reconstruct the transformed image from the plurality of transformed component images.

[0172] The calculation unit is preferably arranged to replace the component image associated with the lowest spatial frequency with a background luminance (e.g. a baseband image) derived from the original image transformed by the apparatus in its fifth aspect. The calculation unit may be arranged to express the contrast of the transformed image as the product of the contrast of the original image and the value of the adjustment factor.

[0173] In a seventh aspect, the present invention may provide an apparatus for transforming an image of a first brightness to adjust its perceived color hue for display by a display device according to a second brightness, the apparatus comprising: a calculation unit for calculating a color adjustment factor for adjusting the color values ​​of the original image; and an adjuster unit for adjusting the color values ​​of the original image according to the color adjustment factor, thereby providing a transformed image for display by the display device according to the second brightness; wherein the calculation unit is arranged to numerically represent the cone photoreceptor response to the color value in view of the corresponding contribution of the rod photoreceptor response to the brightness, and constrain the cone photoreceptor response per unit brightness at the second brightness to substantially match the cone photoreceptor response per unit brightness at the first brightness.

[0174] The contributing rod photoreceptor response is preferably expressed as a brightness-dependent value added to the cone photoreceptor response.

[0175] The calculation unit is preferably arranged to numerically represent the cone photoreceptor response by separately representing the individual responses of the L cone, M cone and S cone in view of the respective corresponding contributions to brightness. The rod photoreceptor response E may be generated by representing R Each cone channel (E L 、E M 、E S ) of the photoreceptor response (L, M, S). For example:

[0176] L=E L +k0E R

[0177] M=E M +k1E R

[0178] S=E S +k2E R

[0179] Here, k i (i=0, 1, 2) is a weighting factor. Preferably, the weighting factor is dependent on the brightness (Y). Preferably, k0=k1. Preferably, k2 is different from k0 and k1. For example, when k0=k1, the value of the weighting factor is dependent on the brightness as follows:

[0180] <![CDATA[Y[cd / m 2 ]]]> 10 0.62 0.10 <![CDATA[k1]]> 0 0.0173 0.173 <![CDATA[k2]]> 0 0.0101 0.357

[0181] The color values ​​are preferably three primary color values.

[0182] In its eighth aspect, the present invention may provide an apparatus for transforming an image having a first brightness to adjust its color saturation for display on a display device having a second brightness, the method comprising: a calculation unit for calculating a color saturation adjustment transform for adjusting color values ​​of an original image; and an adjuster unit for adjusting the color values ​​of the original image according to the color saturation transform. Thus, a transformed image is provided for displaying the display device at the second brightness; wherein the adjuster unit is arranged to: transform the image according to the value of the first brightness (Y) and the second brightness according to the following The color value is adjusted by the value of and the saturation correction factor (s(...)):

[0183]

[0184] The saturation correction factor is a function of brightness, approaches a value of zero as the brightness approaches zero, and gradually and monotonically approaches a value of 1 (1.0) as the brightness increases.

[0185] Color Value Preferably, it is a three-primary color value (eg RGB).

[0186] The calculation unit can be arranged to: determine the average brightness of the original image with the first brightness and determine the average brightness of the image with the first brightness, determine the respective values ​​of the saturation correction factor (s(…)) according to each of the average brightness, and adjust the color value using the respective values ​​of the saturation correction factor.

[0187] In another aspect, the present invention may provide an apparatus for performing the above method.

[0188] In another aspect, the present invention may provide a computer program or a computer program product comprising computer executable instructions arranged to implement the method as described in the above aspects when executed in a computer. The present invention may provide a computer programmed to implement the method as described in the above aspects.

[0189] In another aspect, the present invention may provide a method for adjusting data of an image for display by a display device according to external lighting conditions, the method comprising: providing first brightness data representing a first brightness level of pixels of an image suitable for display under a first external lighting; providing second brightness data representing a brightness level of pixels of the image different from the first brightness data and suitable for display under a second external lighting different from the first external lighting; adjusting the brightness level of the first brightness data so that the image contrast within the entire image represented by the adjusted first brightness data substantially matches the corresponding image contrast within the entire image represented by the second brightness data; determining the background brightness within the entire image represented by the adjusted first brightness data; defining an image sub-region within the image, and adjusting the brightness level of the first brightness data associated with the image sub-region so that the image contrast locally to the image sub-region substantially matches the corresponding image contrast locally to the image sub-region represented by the second brightness data of the image; and using the background brightness of the image sub-region and the adjusted first brightness data to generate brightness image data for use in displaying the image under the second external lighting.

[0190] The step of determining the background brightness may include extracting a baseband of brightness data from the adjusted first brightness data of the entire image.

[0191] Adjusting the brightness level of the first brightness data preferably includes adjusting a tone curve associated with the overall image such that the adjusted first brightness data substantially matches a corresponding image contrast within the overall image represented by the second brightness data.

[0192] The step of extracting the baseband may be performed after adjusting the tone curve and after performing substantially matching image contrast within the entire image.

[0193] The method may include providing a color component associated with the first luminance data. The method may include adjusting the hue of the color component using the first luminance data, the second luminance data, and the color component. The method may also include applying the hue adjustment to the color component to provide, for use, the adjusted first luminance data for the image subregion used in displaying the image under the second ambient lighting.

[0194] The hue adjustment is preferably determined numerically using at least one value representing the response of the cone photoreceptors defined in dependence upon a value representing the response of the rod photoreceptors.

[0195] The step of adjusting the brightness level of the first brightness data associated with the image sub-region may include:

[0196] Decomposing the first data into a plurality of representations of images each at different respective spatial resolutions (eg according to a Laplacian pyramid) Generating luminance image data may comprise replacing the representation of the image with the lowest spatial resolution from among the representations with background luminance.

[0197] Detailed description of the accompanying drawings

[0198] Methods and visual models

[0199] like Figure 2 As shown, the input to the method of the preferred embodiment of the present invention is a scene-referenced image (a high dynamic range image expressed in absolute units) or a display-referenced image in, for example, the sRGB color space.

[0200] In the latter case, it is necessary to transform the image from gamma-corrected pixel values ​​to absolute linear RGB values ​​using a display model, such as the Gamma Offset Gain (GOG) model [Berns 1996]. Similarly, the rescaled results of this method can be transformed back to pixel values ​​using an inverse display model, or alternatively, to the sRGB color space. To model the complex interaction between absolute brightness levels and image appearance, we analyze the problem with respect to three different aspects of the image: global contrast (tone curve), local contrast (details), and color. The following sections discuss each aspect in detail.

[0201] Figure 1 An example of the result of applying the method according to the preferred embodiment is shown. Rescaling from and to a dark display is shown. Figure 1 , left: This is at 2cd / m 2 Peak brightness of the image visible on the display. Figure 1 ,Center: original image. Figure 1 , right: This is for 2cd / m 2 Bright image compensated by the display. When the original image is seen through a neutral density filter (2.0D) that reduces the brightness by a factor of 100, or on a display with the backlight dimmed to 1 / 100 of the original brightness, it will match the appearance of the image on the left. When the image on the right is seen through the same filter, thus simulating a dark display, it will appear similar to the original. Note that the seemingly exaggerated sharpness, color shifts, and brightness changes are not perceived at low brightness levels. The image is best seen when the page is enlarged to ¾ of the screen width and viewed from 0.5m on a 24" monitor.

[0202] Reference Figure 2, schematically showing an apparatus comprising a calculation unit (1) and a transformation unit (9). The calculation unit (1) comprises a global contrast rescaling unit (2) arranged to receive a luminance image (Y) to be transformed and a target luminance with respect to which the input image to be transformed is to say that the contrast and / or color of the input image is to be transformed in order to render a resulting image better for viewing at the target luminance level.

[0203] Optionally, but preferably, a baseband image extractor unit (3) is provided as shown (but may be omitted) and is arranged to receive the output of the global contrast rescaling unit and extract the baseband image therefrom. In this embodiment, the computation unit further comprises a Laplacian pyramid decomposition unit (4) arranged to receive as input the luminance image (Y) to be transformed and to decompose the image into a Laplacian image pyramid comprising a plurality of pyramid levels of different spatial frequency intervals,

[0204] The output of the baseband extractor unit and the output of the Laplacian pyramid decomposition unit are both arranged to be input to a transform unit (9). In a preferred embodiment, the transform unit comprises a pyramid reconstruction unit (5) arranged to reconstruct an image from the pyramid levels received from the pyramid decomposition unit. The pyramid decomposition unit may be arranged to do this using all received pyramid levels except the one having the lowest spatial frequency range and substituting in its place the baseband image passed to it from the baseband extractor unit. In this way, all but one of the pyramid levels plus the baseband image may be used by the pyramid reconstruction unit to reconstruct a transformed image, which may be output for display. In other embodiments, the transform unit may omit the pyramid reconstruction unit and may simply output the globally contrast rescaled image from the global contrast rescaling unit for display.

[0205] In a preferred embodiment, the computation unit further comprises a local contrast rescaling unit (6) arranged to receive as input the Laplacian pyramid image levels output by the Laplacian pyramid decomposition unit and to apply local contrast rescaling thereto as discussed below, and to output the local contrast rescaled image pyramid levels to the pyramid reconstruction unit for reconstruction as described above.

[0206] In some embodiments, the computing unit may (either alone or in combination with the above-described and / or Figure 2The color rescaling unit (together with the other units shown) comprises a color rescaling unit (7) arranged to receive as input the color channels (e.g. RGB) of an image to be transformed according to the color transformation method described herein and to output the result for display. The color rescaling unit is arranged to simultaneously receive the luminance image (Y) either directly output from the global contrast rescaling unit or the (shown) pyramid reconstruction unit or as the original image to be color transformed without undergoing contrast transformation. Alternatively, when both color rescaling and global contrast rescaling are desired, the color rescaling unit may be present in the (shown) transform unit, in which case the output of the global contrast rescaling unit will be input to the color rescaling unit. Alternatively, when both color rescaling and local and / or global contrast rescaling are desired (as shown), the color rescaling unit and the pyramid reconstruction unit may both be present in the transform unit, in which case the output of the pyramid reconstruction unit will be input to the color rescaling unit.

[0207] The input luminance image data and color channel data may initially be "display referenced" data that needs to be adjusted to remove / invert custom characteristics associated with the display device that has provided them (if, in fact, this is the source of the data). In this sense, for example, so-called R'G'B'L' data may be adjusted to RGBL data according to a suitable "forward display model" (8) for subsequent transformation according to the present invention. Once so transformed, it is possible to adjust the RGBL data as needed to take into account the custom characteristics associated with the display device via which the image display is to occur, so that RGBL->R'G'B'L'.

[0208] In this way, we can implement one or both of the global contrast recalibration method and / or the local contrast recalibration and / or color recalibration method of the present invention. Figure 2 For clarity, the case where all recalibration methods are applied in the preferred embodiment is shown.

[0209] Contrast recalibration

[0210] Before discussing the contrast matching model, let us introduce the two measures of contrast that we will use in this section. Michelson contrast is defined as:

[0211]

[0212] Among them, L max and L min is the maximum and minimum brightness values ​​of the sine wave, or alternatively, ΔL is the modulation, L mean is the mean of the sine wave. The Michelson contrast varies between 0 and 1.

[0213] When computing image contrast in a multi-scale representation (such as the Laplacian cone), it is more convenient to use logarithmic contrast:

[0214]

[0215] Logarithmic contrast can be interpreted as the modulation of a sine wave in the logarithmic domain. We will use G and M notation in the rest of the article to distinguish between the two measures. The following formula converts from one contrast ratio to the other:

[0216]

[0217] Our ability to see small contrasts (sensitivity) varies greatly with both the frequency of the stimulus and its brightness. This is demonstrated by a number of contrast sensitivity functions (CSFs) [Barten 1999] (e.g. Figure 3 These changes are best described by the CSF shown in Figure 2. Figure 3 In Figure 2, the contrast sensitivity function (CSF) is shown as it varies with luminance (left) and spatial frequency (right). The function is based on a model from [Mantiuk et al. 2011]. The frequency is given in cycles per degree (cpd).

[0218] The graph shows the change in sensitivity, which is the inverse of the threshold detection contrast. Although the CSF captures essential properties of the visual system, it does not explain the perception of contrast in complex images. This is because the CSF predicts the visibility of very small, almost invisible contrasts that appear on a uniform background, which is atypical for most complex scenes. The change in contrast perception is much smaller for contrasts that are well above the detection threshold. George and Sullivan

[1975] showed this by measuring the magnitude of contrast at one frequency to match the magnitude of contrast at another frequency. They found that the lines of matched contrast across spatial frequencies range from a strongly curved curve for low contrasts, which corresponds most closely to the CSF, to an almost flat line for suprathreshold contrasts. Georgeson and Sullivan coined the expression "contrast constancy" to note the invariance of suprathreshold contrast across viewing conditions.

[0219] There is considerable evidence that contrast constancy holds across a range of frequencies for both narrowband patterns (e.g., sine waves [Barten 1999]) and broadband patterns (e.g., bandpass noise [Brady and Field 1995]). Brady and Field

[1995] reported that contrast matching is almost perfect once the contrast is above the detection threshold, without any gradual transition between near-threshold and supra-threshold vision. However, the same cannot be said for contrast matching across a range of luminances, where significant deviations from contrast constancy can be observed even for relatively large contrast magnitudes [Kulikowski 1976]. Therefore, we need to assume that the contrast constancy mechanism behaves differently in the frequency and luminance domains. Kulikowski

[1976] observed that over a wide range of parameters, two contrast magnitudes match in their appearance when their visual contrasts match. This implies that the physical contrast M minus the detection threshold M t For matching the contrasts must be equal:

[0220]

[0221] Among them, M and It is the Michelson contrast seen at different brightness. The detection threshold M is predicted by the CSF function. t :

[0222]

[0223] where ρ is the spatial frequency in cycles per degree, L a In cd / m 2 = Background luminance in units of . In this consideration, we preferably adopt the CSF from [Mantiuk et al. 2011]. S is the absolute sensitivity factor, which can optionally be used to adjust the absolute threshold for a specific experimental scenario. Using this parameter to adjust the experimental setup, we determined that S = 8.6 produced a good match. 2 The peak sensitivity is M t =0.4%, which is consistent with most CSF measurements.

[0224] Although the Kulikowski model is defined in terms of Michelson contrast, it is convenient to formulate matching contrast in terms of logarithmic contrast:

[0225]

[0226] Note that due to the nonlinear relationship between the contrast measures, Equation 6 is not equivalent to Equation 4. However, if Figure 4As shown, the matching contrast lines are almost identical for both models, except for very high contrast and low brightness. Because there is no data for these high contrast levels, the model cannot be said to be correct or incorrect. We will use logarithmic contrast in this model because it does not suffer from singularities at high contrast.

[0227] Figure 4 This contrast matching model also reveals an important property. Lines matching contrast magnitude are shown as a function of brightness. Lines connect contrast values ​​that should appear the same according to the model. The contrast matching lines are more curved for low contrast, suggesting that low contrast is more affected by brightness than high contrast. This contrast contrast model contrast transducer [Pattanaik et al. 1998; Mantiuk et al. 2008] is in contrast to another popular model of suprathreshold contrast: the contrast transducer [Pattanaik et al. 1998; Mantiuk et al. 2008]. The transducer predicts a much greater increase in physical contrast, regardless of contrast magnitude. This prediction is consistent with experimental data.

[0228] Despite its simplicity, the model proposed by Kulikowski accurately predicts the experimental data. Figure 6 In , we collect contrast matching data from several sources and compare them with model predictions. Figure 6 Contrast matching data (continuous lines) and Kulikowski model predictions from several sources are shown. The different line styles represent the test luminance and reference luminance listed in the legend (in cd / m 2 ) with the contrast at higher luminances plotted on the x-axis. Even when we use the same CSF for all data sets, the model can predict that the physical contrast at low luminances must increase to match the appearance of contrast at high luminances, just as indicated by the amount indicated by the measurements. Kulikowski's model is suitably compared with alternative models of perceptual contrast (e.g., contrast transducers, models of brightness perception, JND luminance scaling) that all form data points very far apart (not included in the plots for better clarity). The model also encompasses our everyday experience of viewing in low light. Objects do not appear blurry at night, as predicted by the multiplicative sensitivity loss in the previous model. Instead, their outlines are sharp, but their textures lose low-contrast detail.

[0229] Global contrast

[0230] The tone curve is a powerful tool for shaping the appearance of an image. It can adjust the perceived contrast in two ways: directly by changing its slope, and in the case of low brightness, indirectly by changing the brightness of the image part and its perceived contrast according to the model from Equation 4. Therefore, the contrast can be increased by using a steeper tone curve (gamma>1), but this tends to make the image darker. Alternatively, a less steep tone curve (gamma<1) can be used to make the image brighter and the perceived contrast higher. In this section, we show how to use the Kulikowski model of matched contrast to solve for the best compromise between two potential solutions.

[0231] Its shape modifies both the physical contrast and the perceived / visual image contrast, where the latter is affected by the absolute brightness. To illustrate this, let us assume that any tone curve can be approximated by a piecewise linear function (e.g. Figure 5 The lower curve shown). Figure 5 In Figure 1, two sliced ​​linear tone curves are shown. The lower curve extends the contrast in bright tones and encompasses the contrast in dark tones. Because the mid-tones are pushed towards lower brightness levels, their perceived contrast will decrease. The opposite is achieved by the upper tone curve. The slope describes the change in physical contrast. If we use a slope of γ = 1.75 to extend the contrast in the brighter tones, we increase both the perceived contrast and the physical contrast for these tones. But this also forces us to compress the darker tones, because the dynamic range of the output display device is limited to the range d min –d max Furthermore, since mid-tones are pushed towards lower luminances, their perceived contrast decreases, as predicted by the model from Equation 4. Therefore, to increase perceived image contrast, an opposite tone curve must be used (e.g. Figure 5 In this section, we show how to find a tone curve that produces optimal perceptual contrast, given the constraints of the output device.

[0232] The task is to find a tone curve T that maps input luminance to output luminance such that the distortion of perceived contrast is minimized. We find this curve with respect to the representative contrast G and spatial frequency ρ. For simplicity, we define the tone curve T in logarithmic luminance space (...)

[0233]

[0234] The resulting physical contrast can be expressed as:

[0235]

[0236] The above formula relies on the fact that the slope of the tone curve in the logarithmic domain corresponds to the contrast change. The problem of finding the optimal tone curve can be expressed as an optimization, where the squared difference between the two sides of the Kulikowski model (Equation 6) is minimized. Formally, it can be expressed as:

[0237]

[0238] obey:

[0239]

[0240] as well as

[0241] T(l min )≥d min , T(l max )≤d max (11)

[0242] G t (l) is the threshold contrast for the log luminance l (Equation 5). The second term of the objective function is the difference between the source (l) and the target log luminance (T(l)) and is weighted by a small constant τ = 0.0001. When the dynamic range of the target image is lower than the dynamic range of the display, this term must push the tone curve towards bright tones or dark tones. The first constraint (Equation 10) ensures that the tone curve is monotonic and increasing. The two remaining constraints (Equation 11) ensure that the tone curve does not exceed the minimum luminance and maximum luminance (d min d max ). Note that the dynamic range and black level of the display are parameters of this method. Therefore, the results can be adjusted for varying contrast ratios of the display and under varying ambient lighting.

[0243] The optional saliency function S(l) is only used for high dynamic range images, which may contain small blocks that greatly extend the dynamic range but do not form a salient part of the image. In this case, it is preferable to choose a tone curve that will best match the appearance of the blocks that form a salient part of the image. This is achieved by assigning weights to different luminance levels during the optimization. In the simplest case, the function is a histogram of the input image (i.e., the weight applied to a given luminance is proportional to or equal to the height of the histogram column for that luminance level within the image's luminance histogram - so more frequent luminance levels receive higher weights), but it is useful to further weight the histogram by a measure of contrast, so that smaller weights are assigned to large, uniform blocks. A disadvantage of using a saliency function is that the tone curve may change between frames. Even with some form of temporal filtering, this can lead to temporal color inconsistencies [Eilertsen et al. 2013]. Therefore, for video processing and display reference scenarios, we preferably set all saliency weights to 1.

[0244] After converting the tone curve into a discrete piecewise linear function, the above optimization problem can be solved numerically efficiently. The quadratic term in the objective function allows us to formulate the problem as a quadratic programming with inequality constraints. Since the threshold function G t Nonlinearity is introduced, so the quadratic problem is preferably solved iteratively, where in each iteration the threshold function is approximated by its first-order Taylor expansion. Since there are very few variables to optimize (typically about 20-30), the solution is efficient. If no significance function is used, the source (l min 、l max ) and purpose (d min d max ) brightness ranges to precompute the solution. For simplicity, we preferably solve the problem for a single representative spatial frequency ρ = 2cpd, which approximately corresponds to the range of brightness levels (cf. Figure 3 - right) and the peak sensitivity of the visual system for a representative contrast G = 0.4. These values ​​were found to produce the best matching results using this experimental setup.

[0245] Figure 7 Several tone curves calculated for different source and target luminance levels are shown in Figure 1. The tone curves are shown for the luminance rescaling that produces the least perceived contrast distortion. The dashed line represents a linear mapping (gamma = 1). Note that when going from 100 to 1 cd / m 2 When recalibrated, the tone curve becomes less sharp (gamma < 1) for bright tones and more sharp for dark tones. This behavior is very different from the typical gamma = 1.5 curve used for "dark" conditions.2 There is also a small change in the shape of the tone curve when recalibrating, since the sensitivity (CSF) does not change much more than 100 cd / m 2 .exist Figure 8 In the top row of can be found the images resulting from the optimized tone curves for different recalibration scenarios.

[0246] Note that in Figure 2 In the preferred embodiment shown, the tone curve is applied to the full-resolution luminance image in a global contrast rescaling step, followed by baseband extraction. This can appear to be more efficient than applying the tone curve to the baseband extracted in the Laplacian pyramid decomposition step. However, this results in strong halo artifacts when a nonlinear tone curve is applied to blurred edges in the baseband image.

[0247] Local contrast

[0248] A well-chosen tone curve can greatly improve the appearance of the rescaled image, however, it provides very coarse control over contrast, limited to the selection of regions of similar brightness. Two other parameters of the contrast matching model are also preferably addressed at the local level: spatial frequency and contrast magnitude. To achieve local contrast control, the preferred embodiment of the present invention uses a Laplacian cone to decompose the image into frequency selective bands (cf. Figure 2 ). The cone can preferably be calculated with respect to the logarithm of the luminance value, so that the bandpass level contains the logarithmic contrast value (Formula 2).

[0249] While spatial frequencies are readily available through multiscale decomposition, estimating the contrast magnitude G requires more care. Contrast in complex images is typically estimated from a bandpass contrast representation that can be extracted from a Laplacian cone [Peli 1990]. However, there are two problems with this approach: a) Contrast is arguably best defined with respect to edges. However, detecting edges requires integrating information across several scales (bands) [Witkin 1984]. Thus, perceived contrast is formed not by a single band, but by integrating information from multiple, or preferably all, bands.

[0250] Sharp edge contrast features decompose into smaller bandpass contrast components at several levels of the pyramid. These bandpass components are smaller than the total edge contrast and will be over-enhanced during rescaling to lower brightness levels, leading to errors in appearance mapping. Figure 9 This is shown visually in . Figure 9 In Figure 2, edges (solid line, top) are enhanced using band-limited contrast (left) or RMS contrast (right) using this local contrast rescaling method. kThe plots for (k=1, 2, 3) show the bandpass contrast or RMS contrast (dashed line) or signal (solid line) in band k after rescaling. Band-limited contrast underestimates the contrast at the edges and results in over-enhancement. RMS contrast can capture the contrast at the edges across the band without over-enhancement. We employ a contrast measure that, in the preferred embodiment, integrates information from multiple (preferably all) frequencies but is localized and captures the contrast of a specific frequency band.

[0251] A common measure of contrast for noise and broadband patterns is the root mean square (RMS) contrast:

[0252]

[0253] Where Y and ΔY are the image brightness and delta at position x, is the mean and the integral is calculated over the entire image. Currently one could use RMS contrast, however, this gives a single value per stimulus and is not very useful for complex images. Therefore, we prefer to adopt a method for localizing the measure by restricting it to a local window (e.g., a Gaussian window). In order to make the calculated contrast measure related to logarithmic contrast, we prefer to operate on the logarithmic luminance image l = log(Y) rather than on the luminance itself. Therefore, the localized broadband contrast can be calculated as:

[0254]

[0255] Among them, * is the convolution operator, g σ is a Gaussian kernel with standard deviation σ. The Gaussian window is preferably arranged to become smaller for higher frequencies to take into account finer scales. This can preferably be achieved by making it equal to half the size of a single cycle at a particular frequency:

[0256]

[0257] Among them, R ppd is the angular display resolution in pixels per visual degree, and ρ is the spatial frequency in cycles per degree. σ is given in pixels assuming a non-decimated Laplace pyramid, where all levels have the same resolution. The frequency ρ can be calculated as:

[0258] ρ=2 -(k+1) R ppd (15)

[0259] where k=1, ..., N are the levels of the pyramid, and k=1 represents the finest level. Given a local contrast estimate, a contrast modification suitable for achieving appearance matching can be expressed as:

[0260]

[0261] Among them, c k (x,y) is the contrast at the pixel location (x,y) and the kth level of the pyramid (Equation 13), where k = 1, ..., N. We choose N so that the coarsest band (other than the base band) has a peak frequency less than or equal to 2 cpd. The function G is the contrast measure transformation given in Equation 3. M t and is the detection threshold for the input and rescaled image (Equation 5).

[0262] To find these thresholds from the CSF, we preferably use the same thresholds as the source (Y) and the rescaled The peak frequency corresponding to the pixel intensity for a given band of the image (Equation 14) is provided by the rescaled baseband image.

[0263] Knowing the necessary modifications, we can perform contrast rescaling as a local enhancement of the Laplacian cone:

[0264]

[0265] Among them, P k Corresponds to the source image pyramid levels. The low-pass baseband is discarded (k=N+1). The resulting image can be reconstructed by summing all modified levels of the pyramid.

[0266] Including baseband, which comes from the global contrast recalibration step (refer to Figure 2 The results of the local contrast rescaling step isolated from the other components of the method can be seen in Figure 8 in the second row. Figure 8 The results produced by different components of the synthesis method in different aspects or preferred embodiments of the present invention are shown. The numbers in the top row indicate the source peak brightness and target peak brightness of the display. Note that the results for rescaling the dark display on the left (100->10 and 100->1) mean that it is visible at much lower brightness levels, despite the neutral density filter shown next to the markers on the top. When viewed through the ND filter, most obvious artifacts (such as haloing and over-sharpening) disappear. Note that the contrast is selectively modified depending on the amount of contrast. This behavior is consistent with how we perceive contrast at different brightness levels.

[0267] Color recalibration

[0268] Reduced brightness affects not only brightness contrast but also color. This is manifested by a loss of color saturation, produced most by a reduced response of the cones and a shift towards more bluish hues, known as the Purkinje shift. The latter effect is attributed to the fact that rods and cones share the same neural pathway to send their signals to the visual cortex [Cao et al. 2008]. In the photopic brightness range, the information from the cones is the dominant signal, while in the mesopic range, the rods become dominant. In the mesopic range, when both types of photoreceptor cells are active, the signals from the rods are combined with those from the cones in an earlier stage of visual processing. The variable contribution of the rod signals to the neural pathways of each cone changes the ratio between the responses, producing a hue shift.

[0269] Given an input linear value [RGB]' and a target brightness The goal is to find the resulting linear Color value.

[0270] We begin by modeling the response of the photoreceptors, which is the spectral contribution L(λ) of the light reaching the retina and the spectral sensitivity σ of each type of photoreceptor, L-, M-, S-cones and rods P The product of (λ):

[0271] E P (C)=∫ λ L(λ)σ P (λ)dλ (18)

[0272] where λ is the wavelength and the exponent P corresponds to the type of photoreceptor: L, M, S, or R. We use the normalized Smith & Pokorny cone basis [Smith and Pokorny 1975] for L-, M-, and S-cone sensitivities, and the CIE 1951 intermediate luminous efficiency function for rods. In general, the incoming light is described as the product of three or more spectral basis functions (π) and their coefficients (p):

[0273]

[0274] Without loss of generality, we can simplify the model and assume that the coefficients P1...3 correspond to linear RGB color values. Figure 10 In Figure 1, we show the spectral basis for several displays we measured. The figure shows the spectral emission of the tested displays. The left plot also shows the Smith & Pokorny cone basis (dashed line), and the right plot shows the CIE intermediate luminous efficiency function (dashed black line). The matrix M used to convert linear RGB values ​​to photoreceptor responses can then be solved E :

[0275]

[0276] Among them, the matrix M E The coefficient of is given by:

[0277] m P，i =∫ λ Π i (λ)σ P (λ)dλ (21)

[0278] Cao et al. [Cao et al. 2008] observed that rod signals share paths with L-, M-, and S-cone signals, and that their effects are additive and depend on the brightness of the signals. The combined response of each cone channel with the rod inputs L, M, and S can be expressed as:

[0279]

[0280] where k1(Y) and k2(Y) are functions that model the rod input strength to L(k1), M(k2), and S(k3) at luminance Y. These functions were obtained by interpolating between the values ​​measured in [Cao et al. 2008] (the value of k2 was scaled by 0.5 due to similar scaling of the S channel response), which is listed in the table below.

[0281] <![CDATA[Y[cd / m 2 ]]]> 10 0.62 0.10 <![CDATA[k1]]> 0 0.0173 0.173 <![CDATA[k2]]> 0 0.0101 0.357

[0282] The signal is then processed further down in the visual cortex and combined into the inverse color space. However, since the transformation to the inverse color space is linear, we can match the colors at this early stage. If the cone contrast relative to the cone response values ​​of two colors is the same, we assume that they will appear similar:

[0283]

[0284] Note that while it is very difficult or impossible to directly match the LMS channels due to the vastly different responses to bright and dark displays, we have found that we can easily match the cone contrast versus cone response. After introducing Equations 20 and 22 into Equation 23, we can solve for the rescaled color values ​​from:

[0285]

[0286] Matching cone contrast allows the present invention to correct for hue shifts in a preferred embodiment. We can also consider the loss of color saturation produced by reducing the sensitivity of the cones and the changes introduced by the tone curve [Mantiuk et al. 2009]. We experimented with the full model of [Cao et al. 2008], which introduces nonlinear gain into the opposing response, but the results were unsatisfactory. The problem arises from the fact that the model does not take into account the display specifications, which causes the results to frequently fall outside the available color gamut if the peak luminance of the two displays is significantly different. Instead, we found that a simple saturation correction works very well. After experimenting with saturation correction in the CIE Lab, CIE Luv color spaces, and a brightness-preserving method [Mantiuk et al. 2009], we found that the best results are produced by the common tone mapping color correction formula:

[0287]

[0288] The same formula applies to the green and blue channels. 2 In the matching experiment of the benchmark image shown, it was found that the matching saturation function s(Y) was obtained.

[0289] The experimental results are shown in Figure 11 , and the best-fit curve is given by:

[0290] s(Y)=Y / (Y+k3)(26)

[0291] where k3 is equal to 0.108. The matching saturation factor is shown in this figure by changing the mean brightness of the image. The black line is the fitted curve (Equation 26). The error spline represents the standard deviation. The results of the color recalibration isolated from the other components of the method can be seen in Figure 8 Note that the hue changes due to the Purkinje shift at low luminances as well as the loss of saturation.

[0292] Summarize

[0293] The method described herein in the preferred embodiment takes as input an image in linear RGB space and the specifications of two displays. A display model is applied to the image to determine its colorimetric properties ( Figure 2). A global tone curve is calculated for the output display specifications (global contrast recalibration step) and applied to the original image. A Gaussian cone is then calculated for this tone corrected image and only its baseband is extracted. The original image is also decomposed into a Laplacian cone and the contrast of each layer except the baseband is modified to match the contrast visible on the original display using Equation 17 (local contrast recalibration step). The baseband of the tone mapped image is then fused with all layers except the baseband of the contrast enhanced Laplacian cone. This produces an image with an improved tone curve, corrected contrast and no halo effect. The color changes caused by the rod input and saturation are estimated based on the input and output luminance, and new linear RGB values ​​are calculated using Equations 24 and 25 (color recalibration step). Finally, the inverse display model of the second display is applied to produce the final image.

[0294] Because human vision does not maintain the same contrast and color perception across a range of luminances, images need to be compensated when displayed at different luminance levels than originally intended. This method can provide such compensation by rescaling night scenes with respect to bright displays or rescaling bright scenes with respect to dark displays. The latter rescaling scenario allows for novel applications where images are compensated with respect to dark displays, which results in significant power savings in mobile devices while maintaining good image quality. While many appearance models and tone mapping operators claim to predict image appearance changes with luminance, we show (see Figure 16 ) None of the existing models consider all relevant effects and fail to produce acceptable results for a range of brightness recalibration scenarios. While typical image appearance models often involve pairing forward and backward perceptual models that differ in the choice of viewing conditions, we take a very different approach with an optimized tone curve. We bring a simple yet powerful contrast matching model to the field of vision science that has not been used in previous image and video applications. The rod contribution to cone vision is used to predict the Purkinje shift and, combined with this new measure, predicts color saturation loss. Each component, as well as the full method, is tested under experimental conditions to ensure a good appearance match.

[0295] application

[0296] Dark display.

[0297] The main application of this method is to compensate for the changes in appearance seen when an image is shown on a much dimmer display. Figure 1 and Figure 8The compensation shown is particularly attractive for mobile devices, which can reduce their backlighting when used in dark environments, thereby reducing power consumption. We found that the peak brightness of a 1000:1 display can be reduced to, for example, 1 cd / m². 2 As small as that. Other brightness reductions result in an excessive loss of color vision that cannot be compensated. It is important to note that compensation can take advantage of new display technologies (such as OLED), which provide very large color gamuts and contrasts. This additional gamut can reproduce the highly saturated colors and contrasts that can be found in the compensated images.

[0298] Age-adaptive compensation.

[0299] Because this method relies on a model of contrast sensitivity, it can be easily extended to account for differences in acuity and sensitivity between young and older observers. Figure 12 In Figure 2, we show the adjusted light intensity of 10 cd / m² for a 20-year-old observer and an 80-year-old observer. 2 Image compensation for peak brightness displays. In this figure, the peak brightness of the display is 10 cd / m² for young and old observers separately. 2 Typically, little compensation is needed for a 20-year-old, but for older viewers detail and brightness must be improved.

[0300] Re-creation of a night scene.

[0301] The method can also recalibrate images of night scenes to reproduce their appearance on much brighter displays. Figure 8 shows an example of recalibration for a test scenario, Figure 13 (center) shows an example for a scene-referenced HDR image. In this latter figure, the optimal exposure from the scene-referenced HDR image (left) is compared with a faithful reproduction of night vision (center) and an exaggerated visualization for a more dramatic effect (right). Compare the differences in the visibility of detail and color. Note that, as expected, the loss of acuity in the church image is only visible in the darker image portions. While several tone mapping operators and appearance models attempt to predict this appearance change, existing methods are unable to accurately predict the full range of the effect, as discussed in the next section.

[0302] Visualization of a night scene.

[0303] The actual appearance change due to low brightness is generally subtle and much smaller than predicted by many vision models. To achieve more dramatic effects in entertainment applications where perceptual accuracy is not critical, it is generally desirable to alter the appearance by a greater degree than predicted by the vision models. Figure 14This is shown in the right image of , where we have adjusted the parameters to show an excessive change in the image appearance.

[0304] Visualization of age-related visual loss.

[0305] Similarly, since it is possible to target dark display compensation with respect to age groups, it is also possible to take age into account when rendering night scenes. Figure 15 In

[15] , we visualized scenes from a driving simulator as visible to 20-year-old and 80-year-old observers. The figure shows a simulation of night vision for 20-year-old and 80-year-old observers. The simulation assumes compensated refraction and age-related visual loss due to reduced retinal illumination (aging miosis and lens aging), glare incompetence, and loss of neural sensitivity. Note the loss of fine details (such as license plate numbers) in the image on the right (when zoomed in on the screen). Driving simulator rendering is a convention of LEPSIS (part of IFSTTAR). To complete the visualization, we included in the application an age-dependent model of glare incompetence based on the CIE recommendation [Vos and van den Berg 1999].

[0306] video.

[0307] When using the content-independent approach (S(l) = 1 in Equation 9), the method does not contain any temporally inconsistent components, and the video can be processed frame by frame. Content-dependent methods require temporal tone curve filtering (such as the one proposed in [Mantiuk et al. 2008]). Examples of recalibrated video clips can be found in the supplementary material.

[0308] Comparison with other methods

[0309] In this section, we compare the proposed method with several alternative techniques.

[0310] CIECAM02 is a state-of-the-art color appearance model that accounts for several brightness-dependent effects (such as the Hun and Stevens effects). To recalibrate images, we process them through a forward CIECAM02 transform and then a reverse CIECAM02 transform. However, we vary a parameter that depends on the viewing conditions between each transform. The viewing conditions vary between dark, dimmed, and average, depending on the source and target brightness levels. We also modify the brightness of the adapted white point to correspond to the drop in brightness levels, but we did not notice a significant impact of this parameter on the results.

[0311] Figure 16 The results of different methods (rows) are shown when rescaling from one brightness level to another (columns). Figure 8 The original image is shown in the lower left corner instead of Figure 8 The 100->10 results for this method can be found in .

[0312] like Figure 16 As shown in the top row of Figure 2, CIECAM02 predicts a loss of perceived contrast and color saturation in low light, and compensates for it by increasing overall image contrast at the expense of reduced brightness (100->1 cd / m 2 As we show later, these images provide an inferior appearance match due to the very low luminance (1->100 cd / m 2 The changes in appearance of the 3D images (in the 3D image) are subtle, confirming that the model is in fact limited to photopic vision.

[0313] Display Adaptive Tone Mapping [Mantiuk et al. 2008] is a tone mapping operator that takes into account the dynamic range and absolute luminance of the target display. This operator utilizes a tone curve optimization similar to the global contrast rescaling method, but based on a transducer function. The operator is limited to a global (spatially invariant) tone curve, which cannot account for frequency-dependent and color effects. We use the original implementation from the pfstools / pfstmo software. Similar to CIECAM02, the algorithm correctly predicts the contrast loss with luminance loss ( Figure 16 (The second row in [ 1 ]). However, due to the transducer function, it over-predicts the effect. In the 100->1 scenario, colors that are too dark to reproduce are clipped to black. This algorithm cannot recalibrate night scenes because it does not take into account the brightness of the input image.

[0314] The Multiscale Model of Adaptation, Spatial Vision, and Color Appearance [Pattanaik et al. 1998] is one of the most comprehensive models of the visual system, accounting for a wide range of appearance phenomena. We reimplemented the algorithm using excerpts of code made publicly available by the authors. Best results are achieved when the low-passband of the target image is multiplied by a constant factor, which is the approach recommended by the authors for images with low dynamic range.

[0315] Figure 16 The results shown demonstrate that the method predicts a wide range of visual phenomena: loss of acuity, Purkinje color shift, loss of color saturation, and loss of contrast. However, it is also clear that the magnitude of all these effects is not correctly predicted: the loss of contrast and acuity due to brightness is excessive, and the color shift due to Punkinje shift is too subtle. The results for 100->1 reveal another limitation shared with most forward-backward vision models: the resulting colors generally exceed the available dynamic range, producing an unreproducible image.

[0316] The calibrated image appearance reproduction model [Reinhard et al. 2012] combines tone mapping and color appearance targets to reproduce images across a range of display devices. We use the authors' implementation and vary the input image brightness and display adaptation based on the source and target brightness levels. The algorithm produces pleasing results on a wide range of high dynamic range images. However, as Figure 16 As shown in the fourth row of , there is little change in image appearance regardless of the recalibration scenario. This suggests that the model does not account for brightness-dependent effects in the non-photopic brightness range.

[0317] The perceptual misadaptation model [Irawan et al. 2005] is a tone mapping operator that can simulate the loss of visibility experienced under changing lighting conditions. Figure 16 As shown in the fifth row of [ ] , this method can predict reduced contrast and brightness for dark scenes. However, it does not include spatial processing that can simulate a loss of acuity, nor does it account for hue and saturation changes. When compensating for a dark display (100->1 scenario), the operator does not produce usable results.

[0318] Tone mapping for low-light conditions [Kirk and O'Brien 2011] uses the same model of Cao et al.

[2008] to simulate the Purkinje shift. However, since this method assumes a complete adaptation of the traversal image to intermediate conditions, it also applies the same process to bright patches visible in photopic vision. The result is Figure 15 The blue mist traversing the image shown in the center. Figure 15 This method is compared to perceptual tone mapping [Kirk and O'Brien 2011] for low-light conditions. The image is the conventional method of Kirk and O'Brien. This method selectively applies a hue shift only in dark areas, producing an image that more closely resembles the perception of a night scene. Kirk et al.'s method also does not simulate acuity loss, loss of cone sensitivity, or changes in perceived contrast.

[0319] This method is the most comprehensive model of the effects of brightness on vision from all the proposed methods. It takes a very different strategy for global contrast recalibration and seeks a tone curve that obeys the constraints of the target display's dynamic range so that the resulting image does not suffer from excessive clipping of pixel values. Color casts due to Purkinje shift are visible, but only at low brightness levels. Local contrast modification does not simply sharpen or blur the image, but selectively reintroduces or removes image details. While larger contrasts are largely unaffected, the loss of acuity leads to the loss of small contrast details. All of these changes produce images that correspond to the actual image appearance when seen in the experimental setup described here.

[0320] Experimental comparison

[0321] To objectively demonstrate that the proposed method provides better appearance matching, we ran paired comparison experiments. From the methods discussed in the previous section, we selected only those that produced acceptable results in a specific recalibration scenario. We included a "gamma" function with an exponent of 1.5 because this is common practice for dark viewing conditions [Fairchild 2005, p. 125]. We also included the original unprocessed image as a control condition. The experimental setup was identical to that described above, except that one part of the screen contained two images that were the result of the two alternative recalibration methods. Depending on the scenario, while wearing a 2.0D filter on one eye or the other, the observer was asked to choose the image that most closely matched the appearance of the image shown to the other eye. Seventeen naive observers who did not participate in the parameter adjustment experiment compared the methods for the eight scenarios using a complete paired design.

[0322] result

[0323] Figure 17 Results of a pairwise comparison experiment scaled in units of JND (higher is better) under the Thurstone Case V assumption are shown, where 1 JND corresponds to a 75% discrimination threshold. Note that the absolute JND values ​​are arbitrary and only relative differences are meaningful. Error bars represent 95% confidence intervals calculated by bootstrapping.

[0324] To estimate which part of the population chooses one method as better than another, the results were scaled in units of JND using a similar approach as in [Eilertsen et al. 2013]. Figure 17 The scaling results in [ ] show that the present method is preferred, as it provides significantly better appearance matching in almost all cases. In only two cases—a portrait image of a woman in the 200->2 scenario and an image of a flower in the 2->200 scenario—does this method compare favorably to the next best, but the ranking is not statistically significant. Surprisingly, very few existing methods provide a better reproduction than the original, unprocessed image.

[0325] When recalibrating for a dark display, even a contrast-enhancing gamma of 1.5 seems to be more harmful than beneficial. Note that we have not included methods that do not work or fail in recalibration scenarios (e.g., display-adaptive TMO in the 2->200 case and misadaptation in the 200->2 case). These results clearly indicate that, unlike existing algorithms, the inventive method can consistently produce good results for two very different recalibration scenarios.

[0326] Derivatives of this method were driven, calibrated, and tested through a rigorous experimental process to ensure good appearance matching across brightness levels. It is important to note that we did not assume the correctness of the visual models from the literature measured for simple stimuli. Instead, we tested them across a range of conditions using complex images. We found that the monocular (haploscopic) matching method gave the most consistent and reproducible results when each eye was adapted to different brightness levels, and was therefore used in all of the present experiments.

[0327] Images were displayed on a colorimetrically calibrated 24" 1920x1200 LCD display (NEC PA241W) and viewed in a dark room. The display was driven at 10 bits per color channel and used a native extended color gamut. A piece of black cardboard was used to separate the display screen into two halves, so that each eye could only see half of the screen. The viewing distance was limited to 85 cm, and the pixel angular resolution was 56 pixels per degree. The observer wore modified welding goggles in which we removed the protective filter from one eye and introduced a photographic neutral density (ND) filter (Kodak Wratten 96 1D and 2D) from the other eye. The choice of eye to cover the filter was randomized between sessions. This setup ensured that the two eyes were separately adapted to different brightness levels and that visual glare did not affect the "darker" eye. The observer was asked to adjust parameters or exercise judgment so that the displayed image shown to the "dark" eye was as close as possible to the reference image shown to the "bright" eye. (Adjustment method).

[0328] At least three expert observers performed each parameter adjustment on 10 images from the Kodak database 1, and the results were averaged. The observers were excluded from the comparison experiments. We used the Powell conjugate direction method [Powell 1964], which is used to minimize a multidimensional function, to iterate over the parameters of the recalibration method. At least two full iterations were completed before the final parameter values ​​were found.

[0329] Citation:

[0330] BARTEN,PGJ1999.Contrast sensitivity of the human eye and its effects on image quality.SPIE Press.

[0331] BARTLESON,C.J.,AND BRENEMAN,E.J.1967.Brightness Perceptionin ComplexFields.Journal of the Optical Society of America 57,7(July),953.

[0332] BERNS,R.S.1996.Methods for characterizing crt displays.Displays 16,4,173-182.

[0333] BRADY,N.,AND FIELD,D.J.1995.What’s constant in contrastconstancy?Theeffects of scaling on the perceived contrast of bandpass patterns.Visionresearch 35,6(Mar.),739-56.

[0334] CAO,D.,POKORNY,J.,SMITH,V.C.,AND ZELE,A.J.2008.Rodcontributions tocolor perception:linear with rod contrast.Vision research 48,26(Nov.),2586-92.

[0335] CHANG,N.,CHOI,I.,AND SHIM,H.2004.DLS:Dynamic backlightluminancescaling of liquid crystal display.IEEE Transactions on Very LargeScaleIntegration(VLSI)Systems 12,8.

[0336] EILERTSEN,G.,WANAT,R.,MANTIUK,R.K.,AND UNGER,J.2013.Evaluation ofTone Mapping Operators for HDR-Video.Computer GraphicsForum 32,7(Oct.),275-284.

[0337] FAIRCHILD,M.D.2005.Color Appearance Models,2nd ed.ed.JohnWiley&Sons.

[0338] GEORGESON,B.Y.M.A.,AND SULLIVAN,G.D.1975.Contrastconstancy:deblurringin human vision by spatial frequency channels.The Journalof Physiology 252,627-656.

[0339] IRANLI,A.,LEE,W.,AND PEDRAM,M.2006.Backlight dimming inpower-awaremobile displays.In Proc.of the Annual Conference on DesignAutomation-DAC’06,ACM Press,New York,New York,USA,604-607.

[0340] IRAWAN,P.,FERWERDA,J.,AND MARSCHNER,S.2005.Perceptuallybased tonemapping of high dynamic range image streams.EGSR.

[0341] KANE,J.P.,AND KURTZ,F.A.,2012.Adapting display color for lowluminanceconditions.Patent app.EP 2526688 A1.

[0342] KEROFSKY,L.,AND DALY,S.2006.Brightness preservation for LCDbacklightdimming.Journal of the Society for Information Display 14,12,1111-1118.

[0343] KIRK,A.G.,AND O’BRIEN,J.F.2011.Perceptually based tone mappingforlow-light conditions.ACM Transactions on Graphics 30,4(July),1.

[0344] KUANG,J.,JOHNSON,G.,AND FAIRCHILD,M.2007.iCAM06:Arefined imageappearance model for HDR image rendering.Journal of VisualCommunication andImage Representation 18,5,406-414.

[0345] KULIKOWSKI,J.J.1976.Effective contrast constancy and linearityofcontrast sensation.Vision research 16,12(Jan.),1419-31.

[0346] MANTIUK,R.,AND SEIDEL,H.-P.2008.Modeling a generic tone-mappingoperator.In Computer Graphics Forum,vol.27,Wiley Online Library,699-708.

[0347] MANTIUK,R.,DALY,S.,AND KEROFSKY,L.2008.Display adaptivetonemapping.ACM Transactions on Graphics(Proc.of SIGGRAPH)27,3,68.

[0348] MANTIUK,R.,MANTIUK,R.K.,TOMASZEWSKA,A.,ANDHEIDRICH,W.2009.Colorcorrection for tone mapping.Computer GraphicsForum(Proc.of Eurographics)28,2,193-202.

[0349] MANTIUK,R.,KIM,K.J.,REMPEL,A.G.,AND HEIDRICH,W.2011.HDR-VDP-2:Acalibrated visual metric for visibility and quality predictions inallluminance conditions.ACM Trans.Graph(Proc.SIGGRAPH)30,4(July),1.

[0350] MORONEY,N.,FAIRCHILD,M.,HUNT,R.,LI,C.,LUO,M.,ANDNEWMAN,T.2002.TheCIECAM02 color appearance model.In Proc.IS&T / SID 10th Color ImagingConference,Society for Imaging Science andTechnology,23-27.

[0351] PATTANAIK,S.N.,FERWERDA,J.A.,FAIRCHILD,M.D.,ANDGREENBERG,D.P.1998.Amultiscale model of adaptation and spatial visionfor realistic imagedisplay.In Proc.of SIGGRAPH’98,287-298.

[0352] PELI,E.1990.Contrast in complex images.Journal of the OpticalSocietyof America A 7,10(Oct.),2032-2040.

[0353] PETIT,J.,AND MANTIUK,R.K.2013.Assessment of video tonemapping:Arecameras S-shaped tone-curves good enough?Journal of VisualCommunication andImage Representation 24,1020-1030.

[0354] POWELL,M.J.1964.An efficient method for finding the minimum ofafunction of several variables without calculating derivatives.Thecomputerjournal 7,2,155-162.

[0355] REINHARD,E.,POULI,T.,KUNKEL,T.,LONG,B.,BALLESTAD,A.,AND DAMBERG,G.2012.Calibrated image appearance reproduction.ACMTransactions on Graphics31,6(Nov.),1.

[0356] SMITH,V.C.,AND POKORNY,J.1975.Spectral sensitivity of the fovealconephotopigments between 400 and 500 nm.Vision Research 15,2,161-171.

[0357] THOMPSON,W.,SHIRLEY,P.,AND FERWERDA,J.2005.A spatial post-processingalgorithm for images of night scenes.Journal of Graphics Tools,1-11.

[0358] VOS,J.J.,AND VAN DEN BERG,T.J.1999.CIE 135 / 1-6DisabilityGlare.Tech.rep.WITKIN,A.1984.Scale-space filtering:A new approachtomultiscale description.In IEEE Int.Conf.on Acoustics,Speech,andSignalProcessing,Institute of Electrical and Electronics Engineers,vol.9,150-153.

Claims

1. A method for transforming an initial image into a target image for display by a display device, the method comprising: A first calculation for calculating a tone curve that maps the brightness levels of the original image to the brightness levels of the transformed image; as well as a first transform for transforming the brightness levels of the original image according to the tone curve to thereby provide a first transformed image for display by the display device according to a peak brightness for display; The first calculation includes determining the tone curve involving an optimization process that optimizes a match between contrast of the original image and contrast of the first transformed image, wherein a peak luminance of the tone curve is less than or equal to a peak luminance of the first transformed image for display by the display device, The method comprises: a fourth transformation for transforming the first transformed image having the first brightness to adjust its perceived color hue for display by the display device according to the second brightness, the fourth transformation comprising: Calculating a color adjustment factor for adjusting the color value of the original image; and The color values ​​of the original image are adjusted according to the color adjustment factor to provide a transformed image for display by the display device at the second brightness, the calculating the color adjustment factor including numerically representing the cone photoreceptor responses to color values ​​given the corresponding contributing rod photoreceptor responses to brightness, and the cone photoreceptor responses per unit brightness at the second brightness are constrained to substantially match the cone photoreceptor responses per unit brightness at the first brightness.

2. The method according to claim 1, wherein The first calculation is performed on a piecewise basis, wherein the tone curve is represented as a piecewise linear function having a plurality of linear segments each representing a specific tonal range, and wherein the determination of the linear segments is performed according to the optimization process.

3. The method according to claim 1 or 2, wherein: The method comprises: a second transform for locally transforming the first transformed image within a sub-block of the first transformed image to adjust image contrast for display by the display device, the second transform comprising: A second calculation for calculating a contrast adjustment factor for adjusting the contrast within the sub-block of the first transformed image; and transforming the contrast within the sub-block of the first transformed image according to a contrast adjustment factor to thereby provide a second transformed image displayed by the display device; wherein the second calculation comprises determining a measure of local contrast within the sub-block and determining therefrom a contrast adjustment factor that optimizes a match between the contrast of the first transformed image and the contrast of the second transformed image within the sub-block, wherein the brightness in the sub-block of the first transformed image does not match the brightness in the sub-block of the second transformed image for display by the display device.

4. The method according to claim 3, wherein: Measuring the local contrast allows for measuring local variations in pixel brightness levels within a defined local area.

5. The method according to claim 3, wherein: The sub-block is defined by a spatial window function centered thereon, wherein the width of the window function is proportional to the inverse of the spatial frequency of the original image, such that for higher spatial frequencies the width of the window function is smaller.

6. The method according to any one of claims 1 to 2, wherein The first computation includes decomposing the original image into an image cone including a plurality of different component images each representing the original image via spatial frequencies within respective ones of a plurality of different spatial frequency bands.

7. The method according to any one of claims 1 to 2, wherein The method comprises: a third transform for transforming the first transformed image having a first brightness to adjust its color saturation for display on the display device having a second brightness, the third transform comprising: calculating a color saturation adjustment transform for adjusting color values ​​of the first transformed image; and adjusting the color values ​​of the first transformed image according to the color saturation transform. Thus, a third transformed image is provided for displaying the display device at the second brightness; wherein, according to the following transformation, the value of the first brightness Y and the second brightness The value of and the saturation correction factor s(...) define the adjusted color value: The saturation correction factor is a function of brightness and approaches a value of zero as the brightness approaches zero, and gradually and monotonically approaches a value of 1.0 as the brightness increases.

8. The method according to claim 3, wherein: The method comprises: a third transform for transforming the second transformed image having the first brightness to adjust its color saturation for display on the display device having the second brightness, the third transform comprising: calculating a color saturation adjustment transform for adjusting the color values ​​of the first transformed image; and adjusting the color values ​​of the second transformed image according to the color saturation transform. Thus, a third transformed image is provided for displaying the display device at the second brightness; wherein, according to the following transformation, the value of the first brightness Y and the second brightness The value of and the saturation correction factor s(...) define the adjusted color value: The saturation correction factor is a function of brightness and approaches a value of zero as the brightness approaches zero, and gradually and monotonically approaches a value of 1.0 as the brightness increases.

9. An apparatus for transforming an initial image into a target image for display by a display device, the apparatus comprising: a first calculation unit configured to perform a first calculation for calculating a tone curve that maps the brightness levels of the original image to the brightness levels of the transformed image; as well as a transform unit for transforming the brightness level of the original image according to the tone curve to thereby provide a first transformed image for display by the display device according to a peak brightness for display; The first calculation includes determining the tone curve involving an optimization process that optimizes a match between contrast of the original image and contrast of the first transformed image, wherein a peak luminance of the tone curve is less than or equal to a peak luminance of the first transformed image for display by the display device, Wherein, the device comprises: Means for transforming the first transformed image having a first brightness to adjust its perceived color hue for display by a display device according to a second brightness, comprising: a second calculating unit, calculating a color adjustment factor for adjusting the color value of the original image; and an adjuster unit that adjusts the color values ​​of the original image according to the color adjustment factor to thereby provide a transformed image for display by the display device at the second brightness, the calculating the color adjustment factor comprising numerically representing cone photoreceptor responses to color values ​​in light of corresponding contributions of rod photoreceptor responses to brightness, and wherein the cone photoreceptor responses per unit brightness at the second brightness are constrained to substantially match the cone photoreceptor responses per unit brightness at the first brightness.

Images