Physiologically based adaptive imaging

Abstract

Method comprising: Receiving an input image comprising pixels in a region that is brightly illuminated; Calculating, with a processor that maps a phenomenological model, an afterimage corresponding to the region, wherein first colors of pixels in the afterimage correspond to an intensity of the brightly lit region; Accumulating the afterimage into an output image, wherein the afterimage brightens the pixels in the region of the output image; Display the output image; Calculating, with the processor mapping the phenomenological model, a second afterimage, wherein at least a portion of second colors of pixels in the second afterimage are different from the first colors; Accumulating the second afterimage into a second output image; and Display the second output image.

Description

FIELD OF THE INVENTION

[0001] The present invention relates to image generation and, more particularly, to generating images based on adaptations of the human visual system. BACKGROUND

[0002] DE 42 35 813 A1 relates to a method for simulating afterglow when displaying digitally generated images on a screen. In this method, m consecutive images are displayed simultaneously on the screen, and with each new image, the oldest image is deleted. For the image structure, m+1 color tones are selected to display the images or the background, and the control signals for generating the color tones, each provided with an address, are stored as a table. Furthermore, for each pixel on the screen, the address corresponding to the desired color tone is stored in a video memory.The afterglow is achieved, for example, by assigning the address of the background color to the pixels of the image to be deleted with each new image in the video memory, and the address of the background color to the pixels of the new image, and the address of the image to be deleted with each new image, while the remaining images retain their assigned address. In the table, the assignment of the colors to the addresses is changed so that the address of the new image is assigned the lightest color, and the address of the remaining images is assigned a darker color, which becomes darker the older the associated image.

[0003] WO 2010 / 024 782 A1 discloses a method for displaying an HDR image on an LDR display device for a user. The method comprises the following steps: (1a) estimating a gaze position of the user by tracking at least one eye of the user, wherein the gaze position of the user is a position on a screen of the LDR device at which the user is looking; (1b) deriving an output image from the HDR image based on the estimated gaze position of the user; and (1c) displaying the output image on the LDR display device.

[0004] In the article by Fain GL et al., "Adaptation in Vertebrate Photoreceptors," Physiological Reviews, Vol. 81, No. 1, January 2001, pages 117-151, aspects of the adaptation of the human optical sensory organs are described.

[0005] The human visual system can operate in a wide range of illumination levels due to various adaptive processes acting in concert. For the most part, these adaptive mechanisms are transparent, leaving the observer unaware of their absolute adaptation status. However, at extreme illumination levels, some of these mechanisms produce perceptible secondary effects or epiphenomena. In bright light, these secondary effects include bleaching afterimages and adaptation afterimages, while in dark conditions, they include desaturation, loss of acuity, mesopic hue shift, and the Purkinje effect.

[0006] Standard displays, such as computer monitors, can only reproduce a fraction of the dynamic range typically encountered in natural environments. Images and videos viewed on conventional displays therefore generally do not trigger the luminance adaptation mechanisms that the visual system regularly undergoes in real-world environments. For a display to realistically reproduce the visual experience of high-dynamic-range (HDR) content, it would be desirable to also reproduce the visual experience associated with adapting to a wide dynamic range.

[0007] One way to address this problem is to design monitors that support larger bit depths and maximum brightness (i.e., luminance), thus shifting the burden of performing adaptation onto the observer's visual system. However, aside from difficulties in turning such prototypes into commercial products, some limitations are inherently intractable; e.g., when very bright objects with proportionally strong radiances are displayed on the screen, the viewer may experience discomfort. Furthermore, such strategies only address the case of very bright scenes, whereas a display that accurately reproduces an extremely low-light scene could only be viewed in total darkness, since any ambient light would prevent the viewer from fully adapting to the display.

[0008] Thus, there is a need to address these problems and / or other problems associated with the prior art. SUMMARY

[0009] This need is addressed by the subject matter of the independent claims. Advantageous embodiments are presented in the dependent claims. A system, computer-readable medium, and method are provided for generating images based on adaptation of the human visual system. An input image is received, an effect-provoking change is received, and an afterimage resulting from a cumulative effect of human visual adaptation is calculated based on the effect-provoking change and a per-photoreceptor-type physiological adaptation of the human visual system. The calculated afterimage may include a bleaching afterimage effect and / or a local adaptation afterimage effect. The calculated afterimage is then accumulated into an output image for display. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 illustrates a flowchart of a method for generating an image based on an adaptation of the human visual system in accordance with one embodiment; Fig. 2A illustrates a black and white input image, an output image, and a local adaptation effect afterimage according to one embodiment; Fig. 2B illustrates a color input image, an output image, and a local adaptation effect afterimage in accordance with one embodiment; Fig. 2C illustrates a graph of per-color component human visual system photoreceptor fading effects over time in accordance with one embodiment; Fig. 2D illustrates fading afterimages over time, which are related to the Fig. 2C, in accordance with one embodiment; Fig. Figure 3A illustrates a prior art input image for inducing a fade afterimage effect; Fig. Figure 3B illustrates bleaching afterimages obtained with the Fig. 3A, at various times, in accordance with one embodiment; Fig. Figure 3C illustrates prior art bleaching afterimages obtained with the Fig. 3A shown input image, at different time points; Fig. 3D illustrates another input image to induce a bleaching afterimage effect, in accordance with one embodiment; Fig. 3E illustrates bleaching afterimages, which are associated with the Fig. 3D shown input image, at different times, in accordance with one embodiment; Fig. Figure 3F illustrates prior art bleaching afterimages obtained with the Fig. associated with the 3D input image at different time points; Fig. Figure 4A illustrates a prior art input image to induce a local adaptation afterimage effect; Fig. Figure 4B illustrates a local adaptation afterimage, which is compared with the Fig. 4A, in accordance with one embodiment; Fig. Figure 4C illustrates part of the Fig. 3A shown input image of the prior art; Fig. 4D illustrates a bleaching afterimage, which is compared with the Fig. 4C, in accordance with one embodiment; Fig. 4E illustrates an output image showing the Fig. 4D and a local adaptation afterimage, which is shown in Fig. 4C, in accordance with one embodiment; Fig. 5A illustrates a flowchart of a method for generating an image based on adaptations of the human visual system in accordance with another embodiment; Fig. 5B illustrates a block diagram of a processing system that generates an image based on adaptations from the human visual system, in accordance with one embodiment; and Fig. 6 illustrates an exemplary system in which the various architecture and / or functionality of the various previous embodiments may be implemented. DETAILED DESCRIPTION

[0010] Secondary effects induced by bright light conditions can induce bleaching afterimages and adaptation afterimages (e.g., local adaptation effects). In dark conditions, secondary effects can include desaturation, loss of sharpness, mesopic hue shift, and the Purkinje effect. Displaying secondary effects explicitly can be used to extend the perceived dynamic range of a conventional computer display. A phenomenological model can be developed for each secondary effect, and computer graphics techniques can be used to render each model. Knowledge of the human visual system can be used to develop phenomenological models to a degree of approximation at which naive observers can mistake the rendered models displayed on a conventional computer monitor for naturally occurring epiphenomena.

[0011] An important property of afterimages is that they track the observer's gaze. Therefore, in one embodiment, a gaze-adaptive display can be used to inject one or more secondary effects into an image scene or images on a conventional computer monitor. A display system can track a user's gaze and adapt the image display accordingly, injecting one or more physiologically motivated artifacts, including adaptation to global light levels, retinal afterimages (both due to adaptation and due to photopigment fading), visual acuity loss in low-light conditions, the Purkinje shift, and the mesopic hue shift into an image for display. The fovea, the area from which we derive most of our visual acuity, does not contain rods.Therefore, in low-light conditions, the central part of our visual field is completely blind; the brain fills it in based on information from neighboring areas. A complete simulation involves locking a blind spot (black disk) to the viewing position, allowing the brain to perform its regular task.

[0012] Fig. 1 illustrates a flowchart of a method 100 for generating an image based on adaptations of the human visual system in accordance with one embodiment. At step 110, an input image is received. In one embodiment, the input image may have different levels of illumination such that one or more regions are brightly lit while other regions of the input image may be dark. At step 120, an effect-provoking change is received. In the context of the present description, an effect-provoking change may be a change in the position of a viewer's view or a stimulus input. A position on a display screen to which a viewer's view is directed may change from a first position to a second position when the viewer's view changes.If the first position is associated with a bright part of the image and the second position with a dimmed part of the image, a fade afterimage effect may result. A stimulus input may include a change between the input image and an output image. For example, a change in global illumination, such as a dimming (e.g., eclipse), or a change in camera position.

[0013] At step 130, an afterimage resulting from a cumulative effect of human visual adaptation is calculated based on the effect-provoking change and a per-photoreceptor-type physiological adaptation of the human visual system. In the context of the present description, a photoreceptor is a rod or a cone of the eye, and a type corresponds to the rod type or cone type (e.g., S, M, and L cone types). Each cone type corresponds to a channel, which may be a channel of a color space, such as the LMS color space. A value in the LMS color space may be converted to the RGB color space. In one embodiment, the calculated afterimage is a bleaching afterimage and / or a local adaptation effect. At step 140, the calculated afterimage is accumulated into an output image for display.In one embodiment, one or more additional secondary effects can be modulated and accumulated into the output image. In one embodiment, the input image can also be accumulated into the output image, such as when the viewer's gaze changes and the camera position does not change. In one embodiment, the output image has a different camera position compared to the input image. In one embodiment, the output image has different global illumination compared to the input image.

[0014] More illustrative information will now be provided regarding various optional architectures and features with which the foregoing framework may or may not be implemented, according to the user's wishes. It should be clearly noted that the following information is provided for illustrative purposes and should not be interpreted as limiting in any way. Any of the following features may be optionally incorporated with or without the exclusion of the other described features.

[0015] Although the focus of the secondary-effect modulation described here is on the extreme ends of the illumination spectrum, a comprehensive analysis encompasses the mechanisms governing adaptation and dynamic-range compression across the full range of luminance. These mechanisms include the pupillary reflex, neural adaptation at various levels of the brain, photoreceptor signal coding around the center, and chemical adaptation of the photoreceptors themselves. The net effect of all these mechanisms is to parameterize a function that maps a large dynamic range in the real world to a smaller range of electrical signals sent elsewhere in the brain—that is, a tonemapping function. The tonemapping function can be used as the basis upon which the modulated secondary effects are applied.

[0016] Perceptually motivated tone mapping functions have been studied extensively. Many modulation techniques share some variation on the Naka-Rushton equation, which predicts the instantaneous response of a photoreceptor to a new stimulus after it has adapted to a particular background illumination. The Naka-Rushton response R(I) is given by R(I)=InIn+σn where I is the luminance of the stimulus, n is a contrast control, and σ is the intensity at which the photoreceptor response is equal to one and a half. In one embodiment, n=0.7. In the log domain, equation (1) represents an S-curve or a smooth step function. The adaptation state is encoded in σ. When a sustained stimulus I¯ is applied, the response will weaken and eventually settle at a value given by a plateau function M(I). The appropriate σ for a background illumination I can be calculated by setting R(I)=M(I) to obtain σ(I¯)=(I¯πM(I¯)−I¯π)1 / π

[0017] The plateau function M(I) can be measured and has a form similar to equation (1).

[0018] Equations (1) and (2) can be treated together as a tonemap curve applied globally to the photopic luminance channel of the image. The final displayed pixel value is J = R(I)γ, where γ is chosen to compensate for nonlinearity in the display response. In one embodiment, γ = 0.455. Color images can be handled by imposing a consistent relationship between color channels before and after the tonemap curve is applied.

[0019] The user's current adaptation state can be represented as the illuminance level A (in log units) to which he or she is adapted. In one embodiment, A is determined by the history of the user's gaze positions and the content of the scene. Assume that the log of the mean scene luminance in a small patch around the user's gaze position represents the target A. T For a timestamp of duration Δt, the updated adaptation state A' is determined by moving from A to A T as follows A'←{A+a1Δt1A<ATA−a2Δt1A> AT where a1 and a2 are the rates of adaptation to brighter and darker stimuli, respectively. In one embodiment, the adaptation rate to brighter stimuli is a2=0.2 and the adaptation rate to darker stimuli is a2=0.75. A model may have separate adaptation rates for two directions (e.g., increasing luminance, decreasing luminance) because the visual system takes longer to adapt to dark environments than to bright ones. In the update step, it would cause A' to reach its target A T in any direction, A' can be equal to A TA' can then be converted to linear luminance I and used in equation (2) to appropriately parameterize the tone mapping curve. The effect of simulated global adaptation, calculated using equations (1), (2), and (3), is to darken the scene as a whole when a relatively bright point is fixated by the user, and to brighten the scene when a relatively dark point is fixated by the user.

[0020] The technique for computing simulated global adaptation described above only computes the expected response of the early visual system when exposed to a particular scene. Ultimately, reproducing the appearance of such a scene in the viewer's visual system is important, which requires compensating for the properties of both the display and the user's real-world visual system. In one embodiment, the global adaptation model makes the simplifying assumption that photoreceptor response and visual appearance are equivalent and is still capable of producing good-quality results (i.e., credible images into which the epiphenomenon models can be injected).

[0021] In an effort to provide a viewer with the perception of viewing radiances brighter than the physical capabilities of the display, synthetic afterimages based on phenomenological models are rendered and displayed on a conventional display. "Afterimage" is a general term used to describe any latent image of a real-world object that persists after the original stimulus ceases. Once induced, they are "locked" to an area of the retina and will therefore move with the viewer's gaze until they completely subside. Afterimages can vary in strength and appearance depending on the nature of the stimulus that induced the afterimage. For example, a short-duration, very bright, color-neutral source, such as a camera flash, may produce a strong cyan afterimage that fades to green and eventually to magenta.Alternatively, fixating on a dark, color-neutral region will increase the sensitivity of the receptor, while fixating on a solid color will induce an image of the opposite color (e.g., a red stimulus will produce a cyan image).

[0022] Bleach afterimages are caused by extremely bright light sources, such as a camera flash. To most clearly observe a bleach afterimage, observe a camera flash from a safe distance (1-2 meters), then quickly close your eyes and cover them with your hand. Depending on the brightness of the flash, you should see a bright green or cyan afterimage that quickly fades to a pink or deep purple color.

[0023] Fig. 2A illustrates a black-and-white input image 200, an output image 205, and a local adaptation effect afterimage 210 in accordance with one embodiment. Adaptation afterimages are generated by long fixations on a steady stimulus. To induce an adaptation afterimage, fixate the cross at the center of the input image 200 for a few seconds and then fixate the cross at the center of the output image 205. One should see a subtle checkerboard pattern with the inverse brightness relationships of the quadrants, similar to the local adaptation effect afterimage 210. Note that the local adaptation effect afterimage 210 is not strictly brighter or darker than the background, but instead is brighter for some parts and darker for others.

[0024] Fig. Figure 2B illustrates a color input image 215, an output image 220, and a local adaptation effect afterimage 225 in accordance with one embodiment. To induce a color adaptation afterimage, fixate the cross at the center of the input image 215 for a few seconds and then fixate the cross at the center of the output image 220. One should see a subtle checkerboard pattern having colors similar to the local adaptation effect afterimage 225 (i.e., a color-opposite afterimage).

[0025] The color in each of the quadrants of the local adaptation effect afterimage 225 is the opposite color of the corresponding quadrant of the input image 215. For example, if a first quadrant 218 of the input image 215 is red, a first quadrant 228 of the local adaptation effect afterimage 225 is cyan. If a second quadrant 219 of the input image 215 is green, a second quadrant 229 of the local adaptation effect afterimage 225 is red. If a third quadrant 216 of the input image is blue, a third quadrant 226 of the local adaptation effect afterimage 225 is green. If a fourth quadrant 217 of the input image 215 is yellow, a fourth quadrant 227 of the local adaptation effect afterimage 225 is magenta.

[0026] The distinct appearance of the two types of afterimages (e.g., bleaching and local-adaptation afterimages) results from a physiological process known as phototransduction. Phototransduction is a process by which photoreceptors convert light into electrical signals. Photoreceptors contain hundreds of thousands of light-sensitive proteins called photopigments. The aggregate behavior of these photopigments determines the photoreceptor response. The photopigment life cycle can explain a specific type of afterimage, called the bleaching afterimage. Shortly after being struck by photons, photopigments enter a bleached state in which they are no longer sensitive to light but continue to contribute to the photoreceptor response. Consequently, the photoreceptor continues to generate a response even after the stimulus has ceased; that is, the photoreceptor generates an afterimage.Eventually, the bleached photopigments are restored to a receptive inactivity state, ending their contribution to the transduction cascade and hence to the afterimage. Similarly, upon exposure to a stimulus, the phototransduction cycle, as well as neural adaptation at different levels of the visual pathway, adapt to the stimulus level; this affects the signals generated by subsequent stimuli. For example, a prolonged bright stimulus will induce lower sensitivity: This lower gain will cause a new, darker stimulus to appear even darker.

[0027] The bleaching afterimages are positive, meaning that they are always brighter than the background on which they are superimposed. However, a bright background can easily mask a bleaching afterimage induced by a stimulus of similar brightness. The appearance of a bleaching afterimage varies over time (as shown in the Fig. 2C and Fig. 2D) and with the radiance of the light source (as shown in Fig. 2D). The time-dependent nature of afterimage phenomena has been well studied and is commonly referred to as the "flight of colors." Importantly, the variation of colors over time and in response to stimulus levels has not been modeled to generate images for display. In one embodiment, the photoreceptor model is represented by declining exponential curves.

[0028] Fig. Figure 2C illustrates a graph 230 of a per-color component human visual system photoreceptor bleaching effect over time 230 in accordance with one embodiment. Bleaching afterimages are caused by light sources bright enough to bleach large portions of photopigments in a photoreceptor. A bright stimulus 232 "bleaches" a portion of photopigments from photoreceptors; these molecules remain bleached even after the stimulus is removed. The graph 230 shows the predicted bleaching levels over time of an L, M, and S cone (shown as lines 236, 235, and 234, respectively), which corresponds to a flash of 1 / 8 second (s), 10 7 Candela / meter 2 (cd / m 2 ), white (which simulates all cone types equally), which ends at time t=0s. As in Fig. As shown in Figure 2C, the responses for each cone type are represented by a different declining exponential curve.

[0029] Fig. 2D illustrates a graph 240 of the predicted appearance of bleaching afterimages over time, which is compared with the Fig. 2C, in accordance with one embodiment. The bleaching levels of the three cone types determine the appearance of a bleaching afterimage. The predicted appearance of the bleaching afterimages over time, shown in graph 240, are estimated in response to short flashes ending at time t=0s, and are shown for a range of stimulus strengths (i.e., flash intensities). As the flash intensity increases, the bleaching afterimages start increasing from cyan 242, always fading to dark magenta 232. The corresponding afterimage color appearance for the bleaching levels shown in Fig. 2C is shown below in the 10 7 cd / m 2 -Line of Fig. 2D. At lower levels of flash intensity, the bleaching afterimages start from magenta 244 and fade to dark magenta 232. The fade pattern shown in graph 240 at a first level of intensity (e.g., between 10 4 and 10 5 cd / m 2 ) may be a result of the relatively strong influence that M-cones have on the final RGB afterimage appearance. In Figure 240, a first tinge of green begins at approximately a second level of intensity (e.g., 10 4,5 cd / m 2 ) to appear. At levels of flash intensity in the graph 240 close to a third level of intensity (e.g. 10 5 and 10 6 cd / m 2 ) the bleaching afterimages start from green 243 and move to dark magenta 232.

[0030] The bleaching level B of a photoreceptor is the fraction of photopigments in the bleached state. The fraction of bleached pigments increases when incident light hits inactive pigments and decreases when bleached pigments are restored to their inactive state. The differential change in the bleaching level induced by a stimulus is predicted by dBdt=b1(1−B)I−b2B where b1 is a bleaching sensitivity parameter, I is the incident luminance, and b2 is the recovery rate of the photoreceptor pigments. The parameters governing the three cone-type bleaching behaviors are selected to produce phenomenologically plausible afterimages. In one embodiment, the LMS photopigment bleaching rate is b1 = 1.2×10 -3 , 1.4×10 -4 , and 5.5×10 -7 for the L, M, and S cones, respectively. In one embodiment, the LMS photopigment bleaching rate b2 = 6.9×10 -1 , 1.4×100 , and 5.5×10 0 for the L, M and S cones, respectively. Equation 4 is solved analytically to find the updated bleaching level B' after a simulation time step of size Δt as B'←(B−B∞)e−(b1I+b2)Δt+B∞ where B ∞ the equilibrium bleaching level assuming a constant stimulus is given by B∞=b1'b1I|b2

[0031] At the equilibrium state for any given light level, a photoreceptor will absorb a portion (B ∞) of its photopigment have bleached. For the model, it is assumed that the contribution of these bleached pigments to the overall photoreceptor response is normally imperceptible and can be considered to be implicitly included as part of equation (1). Therefore, to calculate how visible a bleaching afterimage should be, the deviation from the equilibrium bleaching level is used rather than the absolute bleaching level. In particular, to calculate the linear brightness J for an output pixel, bleaching is added to the preset photoreceptor output. J=R(I)+j1⋅max(B−B∞,0) where j1 is a parameter that controls the strength of the bleaching afterimages. In one embodiment, the bleaching afterimage strength parameter j1 = 0.03. Equation (7) is calculated for each color component of the display color space (e.g., RGB or the like) to account for the different temporal dynamics of the L, M, and S cones. The linear brightness can be modified by a gamma transform for display.

[0032] In practice, equation (7) is calculated separately for each cone type. In one embodiment, the input image is for the LMS color space before equation (7) is calculated. Each type of cone has different bleaching and recovery parameters, which, when converted back to RGB, result in colorful, time-varying afterimages (“flight of colors”). In one embodiment, the matrix H=[1.14−1.180.36−0.081.020.180.30−0.111.25] used to convert the LMS color space values to the RGB space. The appearance of the fade afterimages can be adjusted by modifying b1 and b2 for each photoreceptor type.

[0033] Fig. Figure 3A illustrates a prior art input image 300 that induces a bleaching afterimage effect. The prior art image is from Fairchild, MD, 2008, The HDR Photographic Survey MDF Publications. In particular, the bright area 305 induces a green afterimage effect that darkens and changes to dark magenta over time. The maximum radiance of the input image 300 is approximately 4.5 log cd / m 2 .

[0034] Fig. 3B illustrates bleaching afterimages 310 associated with the Fig. 3A, at different times in accordance with one embodiment. The appearance of bleaching afterimages 310, as predicted by the model, can be simulated by a viewer closing his or her eyes after a brief exposure to the input image 300 (i.e., the scene radiance is lowered to zero). The appearance of the afterimage t seconds after the "blink" is in Fig. 3B. Bleach afterimages 312, 313, and 314 correspond to t=1 / 60s, 0.5s, and 2s, respectively, after the blink. The viewer gaze position is held constant—recall that afterimages move with the viewer gaze position. The scenes are representative of the range of bleach afterimage appearances that the model can generate. In particular, cyan, green, and magenta elements are present in bleach afterimages 312, 313, and 314. In bleach afterimage 312, region 315, which corresponds to region 305 in input image 300, appears green, and the surrounding region is magenta. In the bleaching afterimage 313, the area 320 corresponding to the area 305 in the input image 300 appears as a darker green than the area 315 and the surrounding area is dark magenta.In the bleaching afterimage 314, the area 321 corresponding to the area 305 in the input image 300 appears as a dark magenta and the surrounding area is black.

[0035] Fig. Figure 3C illustrates prior art bleaching afterimages 325 obtained with the Fig. 3A, at different time points. The bleaching afterimages 322, 323, and 324 correspond to t=1 / 60s, 0.5s, and 2s after the blink, respectively. To simulate a realistic afterimage, different temporal dynamics of the L, M, and S cones should be considered in the model. Additionally, the afterimage is calculated after performing a color contrast transform. As a result, the afterimages of a color-neutral stimulus synthesized using the prior art model are also neutral and fail to capture the colorful nature of the bleaching afterimages present in Fig. 3B. In the bleaching afterimage 322, the region 330 corresponding to the region 305 in the input image 300 appears white, and the surrounding region is light gray. In the bleaching afterimage 323, the region 335 corresponding to the region 305 in the input image 300 appears light gray, and the surrounding region is darker gray. In the bleaching afterimage 324, the region 340 corresponding to the region 305 in the input image 300 appears dark gray, and the surrounding region is darker gray and black.

[0036] Fig. 3D illustrates an input image 345 inducing a bleaching afterimage effect, in accordance with one embodiment. Specifically, the bright region 348 (e.g., white flash) induces a green and magenta afterimage effect that darkens and changes to dark magenta over time. The maximum radiance of the input image 345 is approximately 8.5 log cd / m 2 .

[0037] Fig. Figure 3E illustrates bleaching afterimages 350 which are merged with the input image 345 shown in Fig. 3D, at different times in accordance with one embodiment. The appearance of the bleaching afterimages 350 as predicted by the model can be simulated by a viewer closing his or her eyes (i.e., the scene radiance is reduced to zero) after a brief exposure to the input image 345. The appearance of the afterimage t seconds after the "blink" is shown in Fig. 3E. Bleach afterimages 352, 353, and 354 correspond to t=1 / 60s, 0.5s, and 2s, respectively, after the blink. The viewer gaze position is kept constant. These scenes are representative of the range of bleach afterimage appearances that the model can generate. In particular, cyan, green, and magenta elements are present in bleach afterimages 352, 353, and 354. In bleach afterimage 352, region 356, which corresponds to region 348 in input image 345, appears magenta, and the inner region 356 is green. In the bleaching afterimage 353, the area 360, which corresponds to the area 348 in the input image 345, appears as a darker magenta than the area 355, and the inner area 361 is a darker green than the inner area 356. In the bleaching afterimage 345, the area 362, which corresponds to the area 348 in the input image 345, appears as a very dark magenta, and the surrounding area is black.

[0038] Fig. Figure 3F illustrates prior art bleaching afterimages obtained with the Fig. 3D shown input image, at different time points. The bleaching afterimages 372, 373, and 374 correspond approximately to t=1 / 60s, 0.5s, and 2s after the blink, respectively. The different temporal dynamics of the L, M, and S cones are not accounted for in the prior art model. In addition, the afterimage is calculated after performing a color contrast transform. As a result, the afterimages of a color-neutral stimulus synthesized by the prior art model are also neutral and fail to capture the colorful nature of the Fig. 3E. In the bleaching afterimage 372, the region 355 corresponding to the region 348 in the input image 345 appears as light gray, and the inner region 376 is white. In the bleaching afterimage 373, the region 380 corresponding to the region 348 in the input image 345 appears as medium gray, and the inner region 381 is light gray. In the bleaching afterimage 374, the region 383 corresponding to the region 348 in the input image 345 appears as dark gray, and the surrounding region is black.

[0039] Another type of afterimage is a local-adaptation afterimage, which can be attributed in part to the role of calcium ions as a regulatory chemical in transduction. The role of calcium in phototransduction is complex and remains an active area of research. However, a net effect equivalent to simple amplification can be assumed due to the strong correlation between calcium concentration and the overall sensitivity of the cell. The rates of calcium influx and efflux are finite, so changes in the cell's calcium concentration will lag behind a changing stimulus, temporally amplifying or attenuating the photoreceptor response until it reaches a new equilibrium, i.e., generating an afterimage.

[0040] Existing computational models for simulating afterimages do not explicitly distinguish between two types of afterimages, bleaching and local-adaptation afterimages, and therefore fail to reproduce important behaviors. As described further herein, a unified model for bleaching and local-adaptation afterimages can be used. While the appearance of the bleaching and local-adaptation epiphenomena is certainly influenced by higher-level visual processing, in one embodiment the model represents only the photoreceptor level for two reasons: (1) The physiology governing retinal contributions to afterimages is better understood, and (2) a photoreceptor-centric model is sufficient for generating credible afterimages.

[0041] Local adaptation afterimages are caused by exposures to stimuli that last long enough for the individual photoreceptors to adapt locally. Unlike bleaching afterimages, local adaptation afterimages can also be negative in the sense that adaptation reduces the photoreceptor response. Referring back to Fig. 2A, looking at the crosshairs of the output image 205 after fixing the crosshairs in the input image 200 produces a local adaptation afterimage 210, where the quadrants corresponding to the bright quadrants in the input image 200 appear darker. The physiological source of local adaptation is a combination of sensitivity loss due to bleaching (bleached pigment does not respond to light) and varying cellular concentration of calcium ions. The model for local adaptation afterimage effects ignores the effect of bleaching on photoreceptor sensitivity, allowing for decoupling of the parameters controlling bleaching from the parameters controlling local adaptation.

[0042] Calcium enters the photoreceptor through transduction-gated ion channels and is continuously moved out by active pumps. The differential change of intracellular calcium concentration C over time is given by dCdt=c1S−c2C, where c1 and c2 control the inflow and outflow of calcium, respectively, and where S is the fraction of open ion channels in the photoreceptor membrane (which is inversely correlated with the magnitude of the photoreceptor response). In one embodiment, the calcium inflow rate is c1=1 and the outflow rate is c2=1. Directly using Equation (8) can lead to numerical instabilities, so S is instead approximated by 1 - R, where R is from Equation (1). The key difference here is that 1 - R does not include afterimages, whereas S does. In our model, we apply the analytical solution to this differential equation to find the updated calcium concentration C' after a simulation time step of size Δt as follows: C'←(C−C∞)e−c2Δt+C∞, where C ∞ the equilibrium calcium concentration assuming a constant R, i.e. constant stimulus and no global adaptation. C ∞ is given by C∞=c1(1 R)c2

[0043] Similar to the case of bleaching afterimages, we primarily consider the deviations from the equilibrium calcium concentration C ∞ , when determining the appearance of a local adaptation afterimage. We apply a local adaptation gain to the photoreceptor response as calculated by: α=Cmax−C∞Cmax C where C max = c1 / c2, the maximum possible equilibrium calcium concentration (which occurs in total darkness). Note that when C < C ∞ , α < 1 and vice versa. The particular definition of α is chosen because it produces plausible results; other functions that have a similar relationship between C, C ∞ , and ensure α would be similarly useful.

[0044] Just as with bleaching conditions, calcium concentration can be tracked separately for each of the cone types at each pixel. Thus, fixating a strong blue stimulus will decrease the sensitivity of S cones, while L and M cones adapt to become more sensitive. When a neutral gray stimulus is later presented, the S cone response will be attenuated, while the L and M cone responses will be enhanced, resulting in the perception of yellow. Tracking calcium concentration can be a source of a color-opposite feature of a model for local adaptation afterimage effects. The same principle applies to neutral-color stimuli, which can decrease or increase the sensitivities of small areas of the retina.

[0045] Fig. Figure 4A illustrates a prior art input image 400 that induces a local adaptation afterimage effect. A viewer fixates on a bright region of the input image 400, indicated by arrow 405. The prior art image is from Fairchild, MD, 2008, The HDR Photographics Survey, MDF Publications.

[0046] Fig. Figure 4B illustrates an output image 410 into which a local adaptation afterimage, which is identical to the Fig. 4A, in accordance with one embodiment. In one embodiment, the same parameters c1 and c2 ensure that the adaptation afterimage that is computed does not produce unnatural color casts. Long fixations give individual photoreceptors time to adapt. When the gaze moves to a new position, such as a viewer shifting gaze from position 405 of the input image 400 to position 415, the spatially varying photoreceptor sensitivities manifest as an afterimage of the first fixation. When the input image 400 and the adaptation afterimage associated with the bright white lamp in the input image 400 are accumulated into the output image 410, a dark spot 420 appears on the book.

[0047] In contrast, a conventional approach can only produce positive afterimages, yielding unnatural results. For example, a bright object, such as the bright white lamp in the input image 400, can produce an afterimage that is always brighter than the background, so that the area 420 would appear as a bright spot on the book. To combine the bleaching afterimage effect and the local adaptation afterimage effect, equation (7) can be modified: J=aj2⋅R(l)+j1ccmax(B−B∞,0) where j2 controls the strength of the local adaptation afterimages. In one embodiment, the adaptation afterimage strength is j2=0.15. Note that the bleaching afterimage term B is multiplied by the normalized calcium concentration. This modification weakens the appearance of bleaching afterimages in bright situations and enhances them in low-light situations. Fig. Figure 4E visually demonstrates the contribution of this term. Overall, the model combines bleaching simulations and local adaptation afterimages to capture the variations in color and time course.

[0048] Fig. Figure 4C illustrates a prior art input image 430, which is a portion of the Fig. 3A is the prior art input image 300. The prior art image is from Fairchild, MD, 2008, The HDR Photographic Survey, MDF Publications. The enhancement induced by calcium influences the entire transduction process, including cascades initiated by bleached photopigments. The local adaptation afterimages are combined with bleaching afterimages to reflect the induced enhancement.

[0049] Fig. 4D illustrates a bleaching afterimage 440, which is compared with the Fig. 4C, in accordance with one embodiment. Local adaptation is not applied to generate the bleached afterimage 440. A bright white region 435 in the input image 430 appears as a green region 445 surrounded by a dark magenta region in the bleached afterimage 440.

[0050] Fig. 4E illustrates an output image 450 which includes the bleaching afterimage 440 formed in Fig. 4D, and a local adaptation afterimage, which is compared with the one shown in Fig. 4C, in accordance with one embodiment. The local adaptation term C / C max is applied in equation (12) to generate the output image 450.

[0051] The human visual system operates in three different regimes depending on the ambient light level. In brightly lit situations (>10 cd / m 2), the photopic regime is dominant: cones function well, colors are easily distinguished, and spatial visual acuity is maximal. In very dark situations (<10 -3 cd / m 2 ) the scotopic regime is dominant: rods function best, color sensitivity is lost, and spatial acuity is poor. Any illumination level between photopic and scotopic is called mesopic and is characterized by a gradual loss of color sensitivity and acuity.

[0052] Three cues that indicate the darkness level of the scene can be incorporated into a model: the Purkinje effect, the mesopic hue shift, and low-light loss of spatial acuity. Some of these cues have been previously studied in the context of conventional tone mapping. The Purkinje effect shifts the luminous efficiency function from a peak around 555 nm in photopic conditions to a peak near 507 nm in scotopic conditions due to the transition from the working domain of cones to that of rods. The perceptual effect is to change the brightness relationships of blue and red stimuli depending on the overall illumination: a red stimulus rated as "brighter" than a blue stimulus in photopic conditions may appear "darker" in scotopic conditions.Strictly speaking, the Purkinje effect does not refer to any perceived change in color appearance; it refers only to the change in brightness perception.

[0053] The Purkinje effect can be incorporated into a model by modifying the RGB→luminance transform used before applying equation (1). Photopic luminance, for example, is calculated as a weighted sum of RGB values with the highest weight on the R and G channels. Scotopic luminance, on the other hand, places little to no weight on R and favors G and B instead. Mesopic luminance can be defined as a mixture of photopic and scotopic luminance. The interaction between rods and cones in the mesopic regime is an active area of study. The luminous efficiency curve can be measured as a function of both wavelength and adaptation level. For simplicity, a luminous efficiency function is assumed to be a linear interpolation weighted by the normalized log-illumination level ρ, which is given by ρ−L−m2m1−m2 where L is the log-photopic luminance; m1 and m2 denote the upper and lower bounds of the mesopic luminance range in log units, respectively. In one embodiment, the maximum mesopic log-luminance is m1=0 and the minimum mesopic log-luminance is m2=-2. Note that when L ≥ m1, ρ ≥ 1, and when L ≤ m2, ρ ≤ 0. For use in a Purkinje effect model, ρ is fixed to lie between the range [0,1]. The Purkinje effect is difficult to observe and likely has little impact on the perceived realism or brightness of a scene. However, it is a well-documented phenomenon and can be incorporated into a full display system.

[0054] In addition to the Purkinje effect, a loss of color discrimination and a shift in perceived hue accompany mesopic vision, producing a hue shift and desaturation effect. As illumination decreases, the reported color of swatches, including neutral gray, appears to drift toward a faint purple color, with little variation depending on the photopic hue. The ratio between color channels used to reconstruct a color image after global adaptation is replaced by a weighted average of the current color ratio of the scene and the scotopic color ratio of a neutral gray color chip (i.e., a faint purple color), weighted by ρ fixed at [0,1].

[0055] The final cue that can be injected into the system is the loss of spatial acuity in low light. Throughout the mesopic range, spatial acuity decreases linearly with log-luminance. The loss in acuity can be explained by the reduced response of cones and the fact that signals from the many rods are aggregated as part of the input to any given ganglion cell on their way to visual centers of the brain.

[0056] Fig. 5 illustrates a flowchart of a method 500 for generating an image based on adaptations of the human visual system in accordance with another embodiment. Although method 500 is described in the context of a computing system, those of ordinary skill in the art will understand that any system performing method 500 is within the scope of embodiments of the present invention. In one embodiment, a graphics processor is configured to perform method 500. In other embodiments, a central processing unit (CPU) is configured to perform method 500. The method steps are performed by custom logic circuitry, such as a custom processing pipeline. Alternatively, the method steps may be implemented as instructions or microcode for controlling a processing unit.

[0057] At step 510, an input image is received. At step 520, an effect-provoking change is received. In the context of the present description, an effect-provoking change may be a change in the position of a viewer's gaze or a stimulus input. At step 530, a bleaching afterimage effect is calculated based on the effect-provoking change and a per-photoreceptor-type physiological adaptation of the human visual system. In one embodiment, the bleaching afterimage effect produces a negative afterimage in the output image. Equation (7) may be used to calculate the bleaching afterimage effect contribution for each cone type. The bleaching afterimage effect contribution for each cone type may be calculated based on a deviation from the equilibrium bleaching level rather than an absolute bleaching level.The bleaching afterimage effect fraction for each cone type can also vary over time.

[0058] At step 535, a local adaptation effect is calculated based on the effect-provoking change. In one embodiment, the local adaptation effect temporarily decreases the gain in the output image after a bright image is viewed, or increases the gain in the output image after a dim or dim image is viewed. Equation (12) can be used to calculate the local adaptation effect and the bleaching afterimage effect contributions. The contribution due to the local adaptation effect can be calculated based on a deviation from the equilibrium calcium concentration rather than an absolute calcium concentration level, as shown in equation (11). The bleaching afterimage term can be multiplied by a normalized calcium concentration to mitigate the appearance of the bleaching afterimage effect, as shown in equation (12).The local adaptation effect contribution can also vary over time.

[0059] At step 540, the fade afterimage effect and the local adaptation effect are accumulated into an output image to generate the output image for display. In one embodiment, one or more additional effects are accumulated into the output image. For example, the additional effects may include the Purkinje effect, hue shift and desaturation, and spatial sharpness loss.

[0060] At step 545, the processor determines whether another time step is simulated and, if so, returns to step 530 to repeat steps 530, 535, and 540 for another time step to generate an additional output image in a sequence of output images for display. Otherwise, the method terminates.

[0061] A system's viewing environment can affect the quality of the experience. Global adaptation cues are particularly sensitive to ambient lighting. In particular, the illusion of darkness breaks down whenever the global adaptation level A falls below the ambient light level, because if this is done, dark objects are rendered with implausibly bright pixel values. To prevent this from happening, the viewer's ability to globally adapt to any luminance lower than 100 cd / m 2 be artificially limited to roughly correspond to a dimly lit office.

[0062] One consequence of this artificial restriction is that for darker scenes, large portions of the screen are mapped to near-zero pixel values. This often leads to visible quantization banding. To conceal banding, a standard temporal dithering stage can be implemented before the final display.

[0063] Fig. 5B illustrates a block diagram of a processing system 550 that generates an image based on adaptations of the human visual system, according to one embodiment. Processor 555 is configured to execute a model 575 and is coupled to memory 550. Memory 550 may store an input image 565, output image 585, parameters 570 for the model, and an afterimage 580. The processor may execute model 575 to process input image 565 based on parameters 570 to generate afterimage 580. Afterimage 580 may be accumulated into output image 585. The input image may be input image 300, input image 345, input image 400, or input image 430. The afterimage may be one or more of bleaching afterimages 310, 350, and 440. The output image 585 may be one or more of the output images 410 and 450.The bleaching afterimages 310 or 350 may also be considered output images and may be stored as output image 585. The model 575 may be configured to simulate one or more cumulative effects of human visual adaptation. In particular, the model 575 may be configured to simulate bleaching afterimages and local adaptation effects.

[0064] Fig.6 illustrates an exemplary system 600 in which the various architecture and / or functionality of the various previous embodiments may be implemented. As shown, a system 600 is provided that includes at least one central processor 601 connected to a communications bus 602. The communications bus 602 may be implemented using any protocol, such as PCI (Peripheral Component Interconnect), PCI Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communications protocol(s).

[0065] The system 600 also includes input devices 612, a graphics processor 606, and a display 608, e.g., a conventional CRT (cathode ray tube), LCT (liquid crystal display), LED (light-emitting diode), plasma display, or the like. User input may be received from the input devices 612, e.g., keyboard, mouse, touchpad, microphone, gaze tracking, and the like. In one embodiment, the graphics processor 606 may include a plurality of shading modules, a rasterization module, etc. Each of the foregoing modules may even reside on a single semiconductor platform to form a graphics processing unit (GPU).

[0066] In this specification, a single semiconductor platform may refer to a single unitary semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" may refer to multi-chip modules with enhanced connectivity that simulate on-chip operation and provide significant improvements over using a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be located separately or in various combinations of semiconductor platforms according to the user's wishes.

[0067] The system 600 also includes a network interface 604 and may be coupled to a network (e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN), such as the Internet, peer-to-peer network, cable network, or the like) for communication purposes. The system 600 may also include memory 610. The memory 610 may include main memory and / or secondary memory. Control logic (software) and data are stored in the main memory, which may take the form of random access memory (RAM). The secondary memory includes, for example, a hard disk drive and / or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk (DVD) drive, a recording device, or universal serial bus (USB) flash memory.The removable storage drive reads from and / or writes to a removable storage device in a well-known manner. Computer programs or computer control logic algorithms may be stored in the main memory and / or the secondary memory. Such computer programs, when executed, enable the system 600 to perform various functions. The memory 610 and / or any other storage are possible examples of computer-readable media.

[0068] In one embodiment, the architecture and / or functionality of the various previous figures may be implemented in the context of the central processor 601, the graphics processor 606, an integrated circuit (not shown) capable of at least some of the capabilities of both the central processor 601 and the graphics processor 606, a chipset (i.e., a group of integrated circuits configured to operate and be sold as a unit for performing respective functions, etc.), and / or any integrated circuit for that matter.

[0069] Still further, the architecture and / or functionality of the various previous figures may be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system, and / or any other desired system. For example, system 600 may take the form of a desktop computer, laptop computer, server, workstation, game console, embedded system, and / or any other type of logic. Still further, system 600 may take the form of various other devices, including, but not limited to, a personal digital assistant (PDA) device, a cellular phone device, a television, etc.

[0070] While various embodiments have been described above, it should be understood that they have been presented only as a way of example and not by way of limitation. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the exemplary embodiments described above, but should be defined only in accordance with the following claims and their equivalents.

Claims

[1] Method comprising: Receiving an input image comprising pixels in a region that is brightly illuminated; Calculating, with a processor that maps a phenomenological model, an afterimage corresponding to the region, wherein first colors of pixels in the afterimage correspond to an intensity of the brightly lit region; Accumulating the afterimage into an output image, wherein the afterimage brightens the pixels in the region of the output image; Display the output image; Calculating, with the processor mapping the phenomenological model, a second afterimage, wherein at least a portion of second colors of pixels in the second afterimage are different from the first colors; Accumulating the second afterimage into a second output image; and Display the second output image. [2] The method of claim 1, further comprising calculating, with the processor mapping the phenomenological model, a local adaptation effect based on the effect-provoking change. [3] The method of claim 2, further comprising accumulating the local adaptation effect into the output image prior to displaying the output image. [4] The method of claim 2, wherein the local adaptation effect changes over time. [5] The method of claim 1, wherein the region of the input image is associated with a camera flash and the afterimage and the second afterimage are fade afterimages. [6] The method according to claim 1, wherein the input image is accumulated into the output image before the output image is displayed. [7] A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform steps comprising: Receiving an input image comprising pixels in a region that is brightly illuminated; Calculating an afterimage corresponding to the region by mapping a phenomenological model, wherein first colors of pixels in the afterimage correspond to an intensity of the brightly lit region; accumulating the calculated afterimage into an output image, wherein the afterimage brightens the pixels in the region of the output image; Display the output image; Calculating a second output image by mapping the phenomenological model, wherein at least a portion of second colors of pixels in the second afterimage are different from the first colors; Accumulating the second afterimage into a second output image; and Display the second output image. [8] The non-transitory computer-readable storage medium of claim 7, further comprising calculating a local adaptation effect based on the input image by mapping the phenomenological model. [9] The non-transitory computer-readable storage medium of claim 8, further comprising accumulating the local adaptation effect into the output image prior to displaying the output image. [10] The non-transitory computer-readable storage medium of claim 7, wherein the region of the input image is associated with a camera flash and the afterimage and the second afterimage are fade afterimages. [11] The non-transitory computer-readable storage medium of claim 10, wherein the bleaching afterimage effect changes over time. [12] System comprising: a memory configured to store an output image for display and a phenomenological model; and a processor coupled to the memory and configured to: receive an input image comprising pixels in a region that is brightly illuminated; calculate an afterimage corresponding to the region with the processor mapping the phenomenological model, wherein first colors of pixels in the afterimage correspond to an intensity of the brightly lit region; accumulate the afterimage into the output image, wherein the afterimage brightens the pixels in the region of the output image; display the output image; calculate a second afterimage with the processor mapping the phenomenological model, wherein at least a portion of second colors of pixels in the second afterimage are different from the first colors; to accumulate the second afterimage into a second output image; and to display the second output image. [13] The system of claim 12, wherein the region of the input image is associated with a camera flash and the calculated afterimage and the calculated second afterimage are bleaching afterimages.

Images