Display system with dynamic light output adjustment to maintain constant brightness

Abstract

The light output of the display can be dynamically adjusted on the fly. When implemented in a low-persistence display that supports variable refresh rates, this adjustment maintains a constant brightness over a series of frames to eliminate display flicker. When pixel data for a given frame is output to a frame buffer for presenting an image on the display, the time difference between the illumination of the light-emitting elements of the display for the previous frame and the next illumination of the light-emitting elements for the given frame can be determined, and this time difference is used to determine the value of a light output parameter. During the presentation of the image, the light-emitting elements can be illuminated according to the value of the light output parameter. This determination is repeated over a series of frames to dynamically adjust the light output of the display.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This is a PCT application claiming priority to U.S. Patent Application No. 16 / 399,804, filed April 30, 2019, entitled "DISPLAY SYSTEM WITH DYNAMIC LIGHT OUTPUT ADJUSTMENT FOR MAINTAINING CONSTANT BRIGHTNESS," which is incorporated herein by reference in its entirety. [Background technology]

[0002] Traditional displays for virtual reality (VR) systems (such as those embedded in VR headsets) operate at a fixed refresh rate. A display's "refresh rate" is the number of times per second that the display may refresh an image or redraw the screen (e.g., a 90 Hertz (Hz) display refreshes an image 90 times per second). Advances in graphics have led to the development of displays that support variable refresh rates, meaning that the display's refresh rate changes dynamically as frames are rendered. This allows the refresh rate to stay in sync with the varying frame rates from graphics rendering applications such as video games. For example, both G-SYNC™, a technology available from NVIDIA® Corporation, and FreeSync™, a technology available from Advanced Micro Devices®, Inc., provide logic to match the display's refresh rate to the frame rate of a video game. These techniques eliminate screen tearing (e.g., when an image contains some pixels from a previous frame and some pixels from the current frame) and may allow a running video game to target a range of frame rates, instead of targeting a single frame rate, thereby benefiting from the fastest frame rate possible, given the speed of the graphics processing unit (GPU).

[0003] While variable refresh rate techniques are suitable for use with high-persistence displays, where the backlight is on most of the time across a series of frames, these techniques are not suitable for use with VR systems because most VR systems use low-persistence displays, which pulse the backlight on and off at the refresh rate of the display. If a variable refresh rate is used with a low-persistence display, a viewing user will notice the display flickering as the brightness changes with changes in refresh rate. This is because at higher refresh rates, the light pulses occur closer together in time, creating a brightening effect, whereas at lower refresh rates, the light pulses occur farther apart in time, creating a dimming effect. To a user wearing a head-mounted display (HMD), this manifests as a continuous brightening and dimming of the display (i.e., flickering). In practice, low-complexity scenes (e.g., scenes with few moving objects and simple textures) will appear very bright to the user because the refresh rate is likely to be higher for low-complexity scenes, while high-complexity scenes (e.g., scenes with many moving objects and complex textures) will appear very dim to the user because the refresh rate is likely to be lower for high-complexity scenes. Thus, variable refresh rate display techniques have not been suitable for use in VR systems that use low-persistence displays.

[0004] Provided herein are technical solutions for improving and enhancing these and other systems. [Brief description of the drawings]

[0005] The detailed description will now be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features.

[0006] [Figure 1] FIG. 1 illustrates an example technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain constant brightness over a series of frames, according to embodiments disclosed herein.

[0007] [Diagram 2] FIG. 14 illustrates another exemplary technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain constant brightness over a series of frames, according to embodiments disclosed herein.

[0008] [Diagram 3] FIG. 2 illustrates an exemplary timeline for rendering a series of frames with variable frame rate reprojection and presenting corresponding images at a variable refresh rate, according to embodiments disclosed herein.

[0009] [Figure 4] 1 illustrates a flow diagram of an exemplary process for rendering a frame by reprojection on a low persistence variable refresh rate display system, according to embodiments disclosed herein.

[0010] [Diagram 5] 1 illustrates a flow diagram of an exemplary process for dynamically determining values ​​of light output parameters for illuminating a low-persistence variable refresh rate display during presentation of an image, according to embodiments disclosed herein.

[0011] [Figure 6] Exemplary components of a system, including a wearable device such as a VR headset, in which the techniques disclosed herein can be implemented are described. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Described herein are techniques and systems for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames, among other things. The display system may include a display having an array of light-emitting elements (or light sources). As described above, the display system may be a low-persistence display system, meaning that the light-emitting elements emit light for a small portion of the frame time. For example, for a given frame at a 90 Hz refresh rate, the light-emitting elements may illuminate (or emit light pulses) for a duration of approximately 1 millisecond (ms) of the total frame time, with the total frame time being approximately 11.11 ms. In an illustrative example, the display of the display system disclosed herein may be a head-mounted display (HMD) that may be worn by a user. In this example, the display system may be a virtual reality (VR) system or an augmented reality (AR) system. The display itself, such as an HMD, may include one or more display panels that present images based on frames output by a graphics rendering application, such as a video game. These images are viewed by the user through optics included in the HMD, causing the user to perceive the images as if the user were immersed in the VR or AR environment.

[0013] The displays described herein may support variable refresh rates. Thus, a graphics rendering application, such as a video game, may render frames at a variable frame rate, and the refresh rate of the display may be dynamically adjusted to match (or remain synchronized) with the frame rate of the application. In this way, an application may target a range of frame rates, and the refresh rate of the display may be tied to the frame rate of the application, such that a single frame is rendered and a corresponding image is presented with every screen refresh. Because the refresh rate may change dynamically, if the light output of the light-emitting elements remains constant over each light pulse, the brightness of the display may vary over a series of frames, which may be perceived by a user as a flickering display.

[0014] Accordingly, techniques and systems are described herein for maintaining constant brightness of a low-persistence display that supports a variable refresh rate. The logic of the disclosed display system may be configured to output first pixel data to a frame buffer for the purpose of presenting a corresponding first image on the display, the first pixel data being associated with a first frame of a series of frames. The logic may determine a first time difference between an illumination (or light pulse) of a light-emitting element of the display for a preceding frame of the series of frames and a next illumination (or light pulse) of the light-emitting element for the first frame corresponding to the first image being presented. Based on this first time difference (indicative of an instantaneous refresh rate of the display), the logic of the display system may determine a first value of a light output parameter. During the presentation of the first image on the display, the light-emitting element may be controlled to illuminate according to the first value of the light output parameter. This process may be repeated over a series of frames by dynamically determining a value of the light output parameter for each frame, and as a result of varying the light output parameter over the series of frames in this manner, the brightness of the display remains substantially constant over the series of frames, thereby eliminating or at least reducing flicker.

[0015] Generally, when the time difference between a pair of successive light pulses corresponding to a pair of successive frames is relatively short (implying a relatively high refresh rate), the logic of the display system determines a value of the light output parameter that causes a decrease in the magnitude or duration of the light output from the light emitting elements. On the other hand, when the time difference between a pair of successive light pulses is relatively long (implying a relatively low refresh rate), the logic of the display system determines a value of the light output parameter that causes an increase in the magnitude or duration of the light output from the light emitting elements. In this way, a constant brightness is achieved by increasing the magnitude and / or duration of the light output at a relatively low refresh rate and decreasing the magnitude and / or duration of the light output at a relatively high refresh rate.

[0016] As mentioned above, adjusting the light output of the light-emitting elements of the display can be accomplished in multiple ways. For example, the pulse of light for a given frame can be made higher than the height of the previous frame or reference pulse (e.g., by controlling the light-emitting elements to emit light at a higher intensity). In the opposite direction, the pulse of light for a given frame can be made shorter than the height of the previous frame or reference pulse (e.g., by illuminating the light-emitting elements at a lower intensity). Additionally or alternatively, to make the pulse of light higher or shorter, the pulse of light for a given frame can be made wider than the width of the previous frame or reference pulse (e.g., by illuminating the light-emitting elements for a longer duration). In the opposite direction, the pulse of light for a given frame can be made thinner than the width of the previous frame or reference pulse (e.g., by illuminating the light-emitting elements for a shorter duration). To maintain a constant brightness, the area under the pulse (e.g., the area under the square waveform representing the pulse) may remain constant despite dynamic adjustments to the pulse height and / or width (or the representative waveform of the pulse).

[0017] In some embodiments, the reference frame time is associated with a target level of brightness at which the display remains constant. This reference frame time can be used to determine a value of a light output parameter. For example, upon determining the time difference between successive light pulses corresponding to a pair of successive frames, the logic can determine a ratio between the time difference and the reference frame time, and this ratio can be used to determine a value of a light output parameter for adjusting the light output of the light emitting elements for the current frame. The reference frame time may be a time corresponding to an intermediate frame rate between a minimum frame rate and a maximum frame rate of a range of frame rates targeted by the application. For example, if the application targets a range of frame rates from 45 frames per second (FPS) to 144 FPS, the reference frame time used to determine the aforementioned ratio may correspond to a frame time at a frame rate of 90 FPS, which is approximately a frame time of 11.11 ms. This is merely an example, and the reference frame time is configurable based in part on the specifications of the display system in which the techniques described herein are implemented.

[0018] Also described herein are techniques and systems for applying "reprojection" adjustments to low-persistence variable refresh rate display systems, such as display systems including HMDs. The described reprojection techniques can be used to compensate for slight inaccuracies in the prediction of the original pose of the HMD, based on the predicted illumination time for a given frame, predicted before the given frame is rendered. Reprojection can be used to correct pixel data received from an application for a given frame, such as by transforming the pixel data (e.g., through rotation and reprojection calculations) in a manner that takes into account an updated prediction of the pose of the HMD after the actual rendering time of the frame is acknowledged to the display system. Reprojection can be used to mitigate undesirable visual artifacts by presenting the correct pixels at the time the light from the display reaches the user's eyes.

[0019] By dynamically adjusting the light output based on the time difference between successive pulses of light for a pair of successive frames (time difference indicative of instantaneous frame rate), display flicker (i.e., brightness fluctuation) over a series of frames can be reduced, if not eliminated. This can be done independently of or in combination with reprojection. Combining dynamic light output adjustment with the reprojection techniques described herein provides a more robust display system that reduces visual artifacts while also maintaining constant display brightness. In other words, the techniques and systems described herein allow the display system to present the correct scene for where the user is currently looking, and while doing so, maintain constant display brightness.

[0020] The techniques and processes disclosed herein may be implemented by systems such as HMD systems. Also disclosed herein are non-transitory computer-readable media that store computer-executable instructions for implementing the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed in the context of video game applications, specifically VR game applications, as an example, it should be understood that the techniques and systems described herein may provide benefits to other applications, including, but not limited to, non-VR applications (e.g., AR applications), and / or non-game applications, such as industrial machinery applications, defense applications, robotics applications, and the like. Furthermore, the techniques and processes described herein may be implemented in non-HMD systems, such as other display systems that are not considered "near-to-eye" display systems or that do not involve wearable devices. Furthermore, although certain advantages are described for low-persistence display systems that support variable refresh rates, it should be understood that the techniques and processes described herein may be implemented in other types of display systems, and the techniques and processes described herein are not necessarily limited to low-persistence display systems or variable refresh rate display systems.

[0021] FIG. 1 is a diagram illustrating an example technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames, according to embodiments disclosed herein. FIG. 1 depicts a head-mounted display (HMD) 100 worn by a user 102. The HMD 100 is an example of a display system that may implement the techniques and processes described herein. Thus, the HMD 100 may sometimes be referred to herein simply as a "display" or a "display system." The example HMD 100 of FIG. 1 may include a single display panel 104 or multiple display panels 104, such as a left display panel and a right display panel of a stereo pair of display panels. The one or more display panels 104 of the HMD 100 may be used to present a series of image frames (hereinafter referred to as "frames") that are viewable by a user 102 wearing the HMD 100. It should be understood that the HMD 100 may include any number of display panels 104 (e.g., three or more display panels, a pair of display panels, or a single display panel). Thus, the term "display panel," as used in the singular herein, may refer to a single display panel 104 of a display system having any number of display panels (e.g., "display panel" may refer to either display panel 104 of a pair of display panels). For example, in a two-panel HMD 100, a stereo frame buffer may render, for example, 2160×1200 pixels on both display panels of the HMD 100 (e.g., 1080×1200 pixels per display panel).

[0022] The HMD 100 may utilize any suitable type of display technology, such as a light-emitting display that utilizes light-emitting elements (e.g., light-emitting diodes (LEDs)) to emit light during the presentation of frames on the display panel 104. As an example, the display panel 104 of the HMD 100 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0023] The display panel 104 of the HMD 100 may support a variable refresh rate. Thus, the HMD 100 may be operable over any suitable range of refresh rates, such as a range of 45 Hertz (Hz) to 144 Hz, a range of 45 Hz to 120 Hz, a range of 90 Hz to 144 Hz, or any suitable range of refresh rates. The "refresh rate" of a display is the number of times per second that the display may refresh an image or redraw a screen. The number of frames displayed per second may equate to the instantaneous refresh rate of the display. In other words, the logic of the HMD 100 when a series of frames is processed (e.g., rendered) and when the image is presented on the display may target the goal of presenting the image for a single frame of the series of frames per screen refresh.

[0024] The HMD 100 may implement any suitable low persistence driving scheme, including but not limited to a "global flashing" type display driving scheme, or a "rolling band" type display driving scheme. Using a global flashing technique, all light emitting elements 106 (e.g., an M×N array of light emitting elements 106, sometimes referred to herein as a "light source" 106) may be illuminated synchronously to present a single image on the display for a given frame by illuminating all pixels of the display. Using a rolling band technique, individual subsets of light emitting elements 106, and thus individual subsets of pixels, may be illuminated independently and sequentially within a rolling band of illumination during an illumination period. This rolling band technique may be made possible by the light emitting elements 106 being individually addressable. In some embodiments, both the array of pixels and the light emitting elements 106 on the display panel 104 are arranged in rows and columns, but not necessarily one pixel per light emitting element 106 correspondence. In this configuration, for a rolling band type display driving scheme, individual rows and / or individual columns of light emitting elements 106 may be sequentially addressed and / or individual groups of successive rows and / or individual groups of successive columns of light emitting elements 106 may be sequentially addressed. Addressing the light emitting elements 106 in this "rolling" manner results in independently "illuminating" a subset of pixels corresponding to a subset of these individually addressable light emitting elements.

[0025] As used herein, "illuminating a pixel" means illuminating a light-emitting element 106 that corresponds to that pixel. For example, an LCD illuminates a light-emitting element 106 of a backlight to illuminate a corresponding pixel of the display. Furthermore, as used herein, a "subset of pixels" may include an individual pixel or a plurality of pixels (e.g., a group of pixels). Similarly, a "subset of light-emitting elements 106" may include an individual light-emitting element 106 or a plurality of light-emitting elements 106 (e.g., a group of light-emitting elements 106). In some embodiments, a subset of pixels includes a row of pixels, a column of pixels, a group of consecutive rows of pixels, or a group of consecutive columns of pixels. Similarly, a subset of light-emitting elements 106 includes a row of light-emitting elements 106, a column of light-emitting elements 106, a group of consecutive rows of light-emitting elements 106, or a group of consecutive columns of light-emitting elements 106. Thus, in embodiments of the techniques and systems described herein, a subset of pixels may be scanned and illuminated sequentially, such as by sequentially scanning and illuminating each row of pixels starting with the first row of pixels (e.g., the top row of pixels) and ending with the last row of pixels (e.g., the bottom row of pixels), such as by sequentially illuminating a corresponding subset of light-emitting elements. However, any suitable pattern of illumination (e.g., a snake-like pattern of illumination, column-by-column illumination, sequentially multiple rows / columns of pixels or light-emitting elements 106 at a time, etc.) may be used using the techniques and systems described herein.

[0026] To drive the display panel 104, the HMD 100 may include, among other things, a display controller, such as a microcontroller or similar processor, a display driver circuit, and similar electronics for driving the display panel 104. The display driver circuit may be coupled to an array of light emitting elements 106 of the display panel 104 via conductive paths, such as metal traces on a flexible printed circuit. In one embodiment, the display controller may be communicatively coupled to the display driver circuit and configured to provide signals, information, and / or data to the display driver circuit. The signals, information, and / or data received by the display driver circuit may include one or more light output parameters that cause the display driver circuit to illuminate the light emitting elements 106 in a particular manner. That is, the display controller may determine which light emitting elements 106 are illuminated, when the elements 106 are illuminated, the magnitude of the light output (e.g., pulse height), and / or the duration of the light output (e.g., pulse width), etc., and the display controller may communicate the appropriate signals, information, and / or data (e.g., one or more light output parameters) to the display driver circuit to accomplish that purpose.

[0027] The pixel data for a given frame may be output to the HMD 100 or a frame buffer for presenting the frame as an image on the display panel 104 of the HMD 100. The pixel data for each frame may, in some embodiments, include a two-dimensional array of values ​​(e.g., color values) for each pixel. In some embodiments, the pixel data further includes additional data or metadata, such as a depth value. In some embodiments, the pixel data may include data for each pixel represented by a single color and a set of alpha values ​​(e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values ​​for one or more alpha channels). This pixel data may be output to a frame buffer (e.g., a stereo frame buffer) to present a desired visual effect in the image on the display panel 104 of the HMD 100.

[0028] The HMD 100 may represent a VR headset for use in a VR system, such as for use in a VR gaming system. However, the HMD 100 may additionally or alternatively be implemented as an AR headset for use in an AR application. In AR, the user 102 sees virtual objects overlaid on a real-world environment, whereas in VR, the user 102 does not see a real-world environment, but is fully immersed in a virtual environment as perceived through the display panel 104 and optical system (e.g., lenses) of the HMD 100. Although the embodiments described herein relate primarily to a VR-based HMD 100, it should be understood that the HMD 100 is not limited to implementation in a VR application, nor is the display system limited to implementation in the HMD 100.

[0029] In general, a graphics-based application (e.g., a video game) running on a computing device, such as the HMD 100 itself or a computing device (e.g., a personal computer (PC), a game console, etc.) associated with and coupled to the HMD 100 as part of a display / HMD system, may be configured to output a series of frames. The series of frames is ultimately presented as an image on the display panel 104 of the HMD 100. During the presentation of a single image corresponding to a single frame, the low-persistence display system is configured to pulse the light-emitting elements 106 of the display panel 104, such that the user 102 can view the image presented on the display during the pulsing of the lights 108 by emitting light towards the eyes of the user 102. The timeline 110 shown in FIG. 1 illuminates a series of light pulses 108(1), 108(2), ... 108(N) (collectively 108), which may occur at a variable refresh rate of the display system. Each of these light pulses 108 represents an individual emission of light from the light-emitting elements 106 for a given frame. The series of light pulses 108 is offset from, but synchronized with, the rendering of a series of frames by an application due to logic that matches the refresh rate of the HMD 100 to the frame rate of the application that is rendering the frames.

[0030] As illustrated by timeline 110 in Figure 1, the HMD 100 transitions over time from a relatively high refresh rate (towards the left side of the timeline 110) to a relatively low refresh rate (towards the right side of the timeline 110). In an illustrative embodiment, the refresh rate towards the left side of the timeline 110 may be as high as 144 Hz, and the refresh rate towards the right side of the timeline 110 may be as low as 60 Hz. In this regard, timeline 110 in Figure 1 is not to scale, but is merely illustrative of variable refresh rates generally.

[0031] FIG. 1 illustrates a reference frame time 112 (sometimes referred to as a "nominal" frame time) that may be associated with a target level of brightness at which the display remains constant. Consider an example in which the reference frame time 112 is approximately 11.11 ms, corresponding to a refresh rate of 90 Hz. In this example, if the refresh rate remained constant at 90 Hz, the light-emitting elements 106 would be consistently illuminated over a series of frames according to values ​​of light output parameters corresponding to light pulses 108 having pulse duration 114 and pulse magnitude 116, as shown in the center of the timeline 110 of FIG. 1. The light pulses 108 having pulse duration 114 and pulse magnitude 116 at the refresh rate corresponding to the reference frame time 112 achieve a target level of brightness at which the display remains constant. Thus, as the refresh rate changes, light output adjustments are dynamically made to keep the brightness constant at this target level.

[0032] During the relatively high refresh rate toward the left side of timeline 110, and in the time between light pulse 108(1) and light pulse 108(2), logic of the display system may determine a first time difference 118(1) between light pulse 108(1) (i.e., the illumination of light-emitting element 106 for a previous frame in a series of frames) and a next light pulse 108(2) (i.e., the next illumination of light-emitting element 106 for a current frame). Based at least in part on first time difference 118(1), which is indicative of the instantaneous refresh rate, the logic may determine a value of a light output parameter. In the embodiment of FIG. 1, the light output parameter corresponds to or requires a height of light pulse 108(2) for the current frame. This value of the light output parameter, determined based on time difference 118(1), may then be used to control light-emitting element 106 to emit light pulse 108(2) at a particular magnitude. 1 as being smaller than the pulse magnitude 116 associated with the target level of brightness for reference frame time 112. In other words, because the refresh rate toward the left of timeline 110 is higher than the refresh rate corresponding to reference frame time 112 in the center of timeline 110, the height or magnitude of light pulse 108(2) can be adjusted, via light output adjustment 120(1), to a magnitude smaller than the pulse magnitude 116 associated with reference frame time 112. That is, the dashed outline represents a "reference pulse," in pulse magnitude 116 and pulse duration 114, while the solid outline represents the actual light pulse 108(2), and the reference pulse is shown to illustrate the magnitude of light pulse 108(2) relative to reference pulse magnitude 116. Controlling the luminosity of the light output in response to light pulse 108(2) may include driving the light elements 106 with a current and / or voltage lower than the current and / or voltage that may be used to drive the light elements 106 at a refresh rate corresponding to the reference frame time 112.In some embodiments, control of the light output (or light output adjustment 120(1)) is implemented digitally by setting a digitized value to control the light output of the light pulse 108(2).

[0033] During the relatively low refresh rate toward the right side of the timeline 110, and in the time between light pulses 108(N-1) and 108(N), logic of the display system may determine an Nth time difference 118(N) between light pulse 108(N-1) (or the illumination of light emitting element 106 for a previous frame in a series of frames) and the next light pulse 108(N) (or the next illumination of light emitting element 106 for a current frame). Based at least in part on the Nth time difference 118(N), which indicates the instantaneous refresh rate, the logic may determine a value of a light output parameter that may correspond to or require the height of light pulse 108(N) for the current frame. This value of the light output parameter, determined based on the time difference 118(N), may then be used to control light emitting element 106 to emit light pulse 108(N) at a particular magnitude. This particular magnitude of light pulse 108(N) is shown in FIG. 1 as being greater than the pulse magnitude 116 associated with the target level of brightness for the reference frame time 112. In other words, because the refresh rate toward the right of the timeline 110 is lower than the refresh rate corresponding to the reference frame time 112 in the center of the timeline 110, the height or magnitude of light pulse 108(N) may be adjusted, via light output adjustment 120(N), to a magnitude greater than the pulse magnitude 116 associated with the reference frame time 112. Controlling the luminosity of the light output for light pulse 108(N) may include driving the light elements 106 with a higher current and / or voltage than the current and / or voltage that may be used to drive the light elements 106 at the refresh rate corresponding to the reference frame time 112. In some embodiments, control of the light output (or light output adjustment 120(N)) is implemented digitally by setting a digitized value to control the light output of light pulse 108(N). In particular, the display driver circuitry has sufficient (or excess) bandwidth to provide the ability to increase the pulse height.

[0034] In some embodiments, the reference frame time 112 is used to determine a value of a light output parameter that corresponds to the magnitude of the light pulse 108 shown in FIG. 1. For example, the logic of the display system may determine a ratio of the first time difference 118(1) to the reference frame time 112 and use this ratio to determine the value of the light output parameter for the light output adjustment 120(1) associated with the light pulse 108(2). Consider an example in which the first time difference 118(1) is approximately 8.3 ms, corresponding to a refresh rate of 120 Hz, and the reference frame time 112 is approximately 11.11 ms, corresponding to a refresh rate of 90 Hz. In this example, the following ratio may be calculated:

number

[0035] Consider an example where the Nth time difference 118(N) is approximately 16.66 ms, corresponding to a refresh rate of 60 Hz, and the reference frame time 112 is still 11.11 ms. In this example, the following ratio can be calculated:

number

[0036] 2 is a diagram illustrating another exemplary technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames, according to embodiments disclosed herein. As with the example of FIG. 1, as illustrated by the timeline 210 of FIG. 2, the HMD 200 transitions over time from a relatively high refresh rate (toward the left side of the timeline 210) to a relatively low refresh rate (toward the right side of the timeline 210). In the illustrative example, the refresh rate toward the left side of the timeline 210 may be as high as 144 Hz, and the refresh rate toward the right side of the timeline 210 may be as low as 60 Hz. In this regard, the timeline 210 of FIG. 2 is also not to scale, but is merely illustrative of variable refresh rates in general.

[0037] 2 again shows a reference frame time 212 that may be associated with a target level of brightness at which the display remains constant. A light pulse 208 having a pulse duration 214 and pulse magnitude 216 at a refresh rate corresponding to the reference frame time 212 achieves the target level of brightness at which the display remains constant. Thus, as the refresh rate changes, light output adjustments are made dynamically to keep the brightness constant at this target level.

[0038] During the relatively high refresh rate toward the left side of timeline 210, and in the time between light pulse 208(1) and light pulse 208(2), logic of the display system may determine a first time difference 218(1) between light pulse 208(1) (i.e., the illumination of light emitting element 206 for a previous frame in the series of frames) and an upcoming light pulse 208(2) (i.e., the upcoming illumination of light emitting element 206 for a current frame). Based at least in part on first time difference 218(1), which is indicative of the instantaneous refresh rate, the logic may determine a value of a light output parameter. In the example of FIG. 2, the light output parameter corresponds to or requires a width of light pulse 208(2) for the current frame. This value of the light output parameter, determined based on time difference 218(1), may then be used to control light emitting element 206 to emit light pulse 208(2) for a particular duration. This particular duration of light pulse 208(2) is shown in FIG. 2 as being shorter than the pulse magnitude 214 associated with the target level of brightness for reference frame time 212. In other words, because the refresh rate toward the left of timeline 210 is higher than the refresh rate corresponding to reference frame time 212 in the center of timeline 210, the width or duration of light pulse 208(2) may be adjusted, via light output adjustment 220(1), to a duration less than the pulse duration 214 associated with reference frame time 212. Controlling the duration of the light output for light pulse 208(2) may include driving light elements 206 for a shorter amount of time than the amount of time that light elements 206 could be driven at the refresh rate corresponding to reference frame time 212. In some embodiments, control of the light output (or light output adjustment 220(1)) is implemented digitally by setting a digitized value to control the light output of light pulse 208(2).

[0039] During the relatively low refresh rate toward the right side of the timeline 210, and in the time between light pulses 208(N-1) and 208(N), logic of the display system may determine an Nth time difference 218(N) between light pulse 208(N-1) (or the illumination of light emitting element 206 for a previous frame in a series of frames) and the next light pulse 208(N) (or the next illumination of light emitting element 206 for a current frame). Based at least in part on the Nth time difference 218(N), which indicates the instantaneous refresh rate, the logic may determine a value of a light output parameter that may correspond to or require a width of light pulse 208(N) for the current frame. This value of the light output parameter, determined based on the time difference 218(N), may then be used to control light emitting element 206 to emit light pulse 208(N) for a particular duration. This particular duration of light pulse 208(N) is shown in FIG. 2 as being longer than the duration 214 of the pulse associated with the target level of brightness for the reference frame time 212. In other words, because the refresh rate toward the right of the timeline 210 is lower than the refresh rate corresponding to the reference frame time 212 in the center of the timeline 210, the height or magnitude of light pulse 208(N) may be adjusted, via light output adjustment 220(N), to a width or duration that is greater than the duration 214 of the pulse associated with the reference frame time 212. Controlling the duration of the light output for light pulse 208(N) may include driving light elements 106 for an amount of time that is longer than the amount of time that the light elements 206 could be driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, control of the light output (or light output adjustment 220(N)) is implemented digitally by setting a digitized value to control the light output of light pulse 208(N). In particular, the display driver circuitry has sufficient (or excess) bandwidth to provide the ability to increase the width of the pulse.

[0040] Again, the reference frame time 212 can be used to determine a value of a light output parameter that corresponds to the duration of light pulse 108 shown in FIG. 2. For example, the display system logic can determine a ratio of first time difference 218(1) to the reference frame time 212 and use this ratio to determine a value of a light output parameter for light output adjustment 220(1) associated with light pulse 208(2). Consider an example where first time difference 218(1) is approximately 8.3 ms, corresponding to a refresh rate of 120 Hz, and reference frame time 212 is approximately 11.11 ms, corresponding to a refresh rate of 90 Hz. In this example, the following ratio can be calculated:

number

[0041] Consider an example where the Nth time difference 218(N) is approximately 16.66 ms, corresponding to a refresh rate of 60 Hz, and the reference frame time 212 is still 11.11 ms. In this example, the following ratio can be calculated:

number

[0042] In some embodiments, each of the approaches shown in Figures 1 and 2 may be combined. For example, the magnitude and duration of the light pulse 108 may be dynamically adjusted by determining a first value of a first light output parameter corresponding to the magnitude of the light pulse 108 / 208 and determining a second value of a second light output parameter corresponding to the duration of the light pulse 108 / 208, and illuminating the light-emitting elements according to both the first and second values ​​to adjust the light output.

[0043] In embodiments where the display system uses a rolling band type display driving scheme, the light output may be adjusted at least in part by changing the thickness of the rolling band that traverses the display panel 104 / 204 during the illumination period of a given frame. That is, the value of the light output parameter may correspond to a number of light emitting elements that are simultaneously illuminated (e.g., a number of rows that make up the rolling band of illumination). A display system using a global flashing type display driving scheme may adjust the duration of the light pulse 108 / 208, while a rolling band, which is equivalent to adjusting the duration of the light pulse, may adjust (e.g., increase or decrease) the number of light emitting elements that are simultaneously illuminated during the illumination period. A "thicker" rolling band may effectively shorten the duration of the light pulse, while a "thinner" rolling band may effectively lengthen the duration of the light pulse.

[0044] It should be appreciated that determining the time difference 118 / 218 between a pair of consecutive light pulses 108 / 208 corresponding to a pair of consecutive frames may include determining the time difference between any two corresponding points between the pair of light pulses 108 / 208 or the pair of frames. For example, the time difference 118 / 218 may be determined between the start of each light pulse 108 / 208, the midpoint of each light pulse 108 / 208, the end of each light pulse 108 / 208, etc. The display system logic, in some embodiments, may include frame start markers in the pixel data between frames rendered by each application, and these frame start markers may be used to determine the time difference for purposes of determining values ​​of the light output parameters.

[0045] FIG. 3 illustrates example timelines 310(1) and 310(2) for rendering a series of frames 302 with variable frame rate reprojection and presenting corresponding images at a variable refresh rate, according to embodiments disclosed herein. The example of FIG. 3 illustrates three example frames 302(1) (or frame "F"), 302(2) (or frame "F+1"), and 302(3) (or frame "F+2") with respect to a rendering timeline 310(1) to illustrate how the frames 302 may be rendered serially, where an application 304 renders first frame F, then frame F+1, then frame F+2, sequentially from left to right on the rendering timeline 310(1). The rendering timeline 310(1) also illustrates the rendering workload 306 of a compositor 308 of an HMD 100 / 200 (or display system) toward the end of each rendering interval for each frame 302. Application 304 spends a certain amount of time rendering each frame 302. This amount of time is shown in Figure 3 as application rendering time 312. Because frame rates may vary, application rendering time 312(1) for frame 302(1) (or frame "F"), application rendering time 312(2) for frame 302(2) (or frame "F+1"), and application rendering time 312(3) for frame 302(2) (or frame "F+2") may be different amounts of time. Figure 3 shows an example in which application rendering time 312 progressively increases from frame F to frame F+2.

[0046] The individual rendering workload 306 of the compositor 308 for a given frame 302 may represent adjustments applied to pixel data output by the application 304 before rendering the final image on the HMD 100 / 200. The rendering workload 306 may be approximately 1%-5% of the rendering workload of the application 304, which means that it typically takes the application 304 much longer to render the frame 302 than it would take the compositor 308 to apply adjustments to the pixel data output by the application 304 during the rendering workload 306 of the compositor 308. Such adjustments may include, but are not limited to, adjustments for chromatic aberration, panel masking, reprojection, etc., that are applied to the frame 302 output by the application 304 before rendering the final image on the HMD 100 / 200. The application 304 may represent a video game application, or any other type of graphics-based application. The application 304 may execute in a graphics pipeline that outputs pixel data, and the compositor 308 is configured to modify the pixel data and output the modified pixel data to a frame buffer (e.g., a stereo frame buffer).

[0047] Before an application 304 begins rendering a given frame 302, logic in the display system (e.g., compositor 308) may determine a predicted illumination time 314 for the given frame 302, which represents the time that light-emitting elements 106 / 206 will illuminate for the given frame 302. Actual illumination times 314(1), 314(2), and 314(3) for frames 302(1), 302(2), and 302(3), respectively, are shown on a “scanout+illumination” timeline 310(2). The compositor 308 (or other logic) is tasked with predicting this illumination time 314 on a frame-by-frame basis, which is variable due to the variable frame rate and variable refresh rate. The prediction of illumination time 314 by the compositor 308 may be based on past application rendering times 312. For example, the compositor 308 can look at the application rendering time 312 of a previous frame (frame F-1, not shown in FIG. 3) and assume that the application rendering time 312(1) of frame F is the same as the previous frame. The compositor 308 can add a fixed scanout time 316 to the predicted application rendering time 312 to determine the predicted illumination time 314.

[0048] This predicted lighting time 314 is used together with head tracking data generated by the head tracking system of the HMD 100 / 200 to determine a predicted pose that the HMD 100 / 200 will assume at the predicted lighting time 314 of a given frame 302. The compositor 308 sends pose data indicative of this predicted pose to the application 304 for rendering the frame 302, and receives pixel data for the frame 302 from the application 304 when rendering of the frame 302 is finished. By providing the pose data to the application 304 in advance, the application 304 can output pixel data for rendering an image on the HMD 100 / 200 in a manner that is correct for the predicted pose of the user's 102 / 202 head at the future predicted lighting time 314. This means that the application 304 renders a scene appropriate for the predicted pose of the user's head at the lighting time 314 when light from the display panel 104 / 204 reaches the user's 102 / 202 eyes. In the example of FIG. 3, after frame F is rendered by application 304, compositor 308 may determine an actual application rendering time 312(1) for frame F, update the compositor 308's prediction for illumination time 314, and determine a new predicted pose for HMD 100 / 200 based on this updated prediction for illumination time 314. Based on the original predicted pose of HMD 100 / 200 and the new predicted pose of HMD 100 / 200 (e.g., by comparing the two sets of pose data), compositor 308 may determine reprojection adjustments and apply these reprojection adjustments to the pixel data during rendering workload 306(1) to modify the pixel data for frame F. The modified pixel data for frame F is output to a frame buffer and scanned out during scan-out time 316(1) to present an image corresponding to frame F (modified with the reprojection adjustments) on display panel 104 of HMD 100 / 200 at illumination time 314(1).During scan-out time 316, a subset of pixel values ​​(pixel data) of frame F are sequentially scanned out to the display panel 104 via a display port (e.g., High-Definition Multimedia Interface (HDMI®)). During illumination time 314, the light elements 106 / 206 of the display panel 104 are illuminated to illuminate the pixels. The illumination may be a global flashing where all light elements 106 / 206 are performed simultaneously, or a rolling illumination where a subset of the light elements 106 / 206 are performed sequentially. In either case, illumination is considered herein as a pulsing of the lights 108 / 208 for a given frame.

[0049] The graphics logic of the HMD 100 / 200 (or display system) may be asynchronous or synchronous. In an asynchronous system, the compositor 308 runs separately (on a separate asynchronous thread) from the application 304 on the graphics processing unit (GPU) of the HMD 100 / 200 (display system or HMD system). For example, the application 304 may call a function to receive pose data from the compositor 308, which may provide the application 304 with the requested pose data (predicted for the predicted illumination time 314(1) of frame F) so that the application 304 can render frame 302(1) (e.g., frame F) according to the pose data corresponding to the virtual camera pose used to render the scene. Because a variable refresh rate is supported, the compositor's workload 306 starts after the application 304 finishes rendering frame 302 (application rendering time 312 is variable). The compositor 308 may be configured to receive frames 302 (e.g., left and right images) from the application 304 and warp the frames 302 into a back buffer on the display panel 104 / 204 of the HMD 100 / 200. During this workload 306 of the compositor 308, reprojection adjustments may be applied.

[0050] By dynamically matching the refresh rate to the frame rate of the application 304, a newly rendered frame 302 is received from the application 304 at every screen refresh, and an image corresponding to the given frame 302, if present, can be presented with only a small degree of reprojection adjustment. If the predicted application rendering time 312 is accurate, the extent of the reprojection adjustment is minimized. Even with an inaccurate prediction of the application rendering time 312, and thus an inaccurate pose prediction, due to the variable refresh rate capabilities of the display system, the amount of reprojection is small, which means that undesirable visual artifacts caused by reprojection for moving or animated objects are mitigated.

[0051] The processes described herein are described as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or combinations thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement a process.

[0052] 4 illustrates a flow diagram of an exemplary process 400 for rendering a frame by reprojection in a low-persistence variable refresh rate display system according to an embodiment disclosed herein. For discussion purposes, process 400 is described with reference to the previous figure.

[0053] At 402, a frame 302 of the series of frames may be rendered to present a corresponding image on a display panel 104 / 204 of the HMD 100 / 200. For discussion purposes, this frame 302 is referred to as a first frame 302(1). If the HMD 100 / 200 includes a pair of display panels, the first frame 302(1) may be rendered to present a first image on the left display panel 104 / 204 and a second image on the right display panel 104 / 204, or vice versa. As indicated by the various sub-blocks in 402, there may be various sub-operations performed by logic of the HMD 100 / 200 (or HMD system) to render the first frame 302(1) at block 402.

[0054] In sub-block 404, logic of the HMD 100 / 200 (or HMD system) (e.g., logic including the compositor 308) may determine a predicted illumination time 314(1) for the first frame 302(1). Prior to predicting this illumination time, the application 304 may make a call to the compositor 308 to determine the pose of the HMD 100 / 200 so that the application 304 can properly render the first frame 302(1) for that HMD pose. The predicted illumination time 314 determined in sub-block 404 may represent the time that the light-emitting elements 106 / 206 will illuminate for the first frame 302(1). In other words, the prediction of the illumination time in sub-block 404 is a prediction of when light will actually reach the eyes of the user 102 / 202 when an image corresponding to the first frame 302(1) is presented on the display panel 104 / 204 of the HMD 100 / 200. As indicated by sub-blocks 406 and 408 within sub-block 404, there may be various sub-operations performed by logic of the HMD 100 / 200 (or HMD system) for the purpose of determining the predicted illumination time 314(1) in sub-block 404.

[0055] In sub-block 406 , logic of the HMD 100 / 200 (or HMD system) (e.g., logic including the compositor 308 ) may determine a first amount of time that the application 304 spent rendering the previous frame 302 .

[0056] In sub-block 408, the logic may determine a predicted rendering time 312(1) representing an amount of time that the application 304 will spend rendering the first frame 302(1) based at least in part on the first amount of time determined in sub-block 406 (because the application rendering time 312 is variable over a series of frames 302, depending on the complexity of the scene and / or the current load on processing resources, etc.). In one example, if the application 304 spent 11 ms rendering the previous frame 302, then the predicted rendering time 312(1) determined in sub-block 408 may be the same amount of time, 11 ms. This is based on the idea that it is reasonably safe to assume that the next frame 302 will take about the same amount of time to render as the previous frame 302 took to render. The predicted rendering time 312(1) determined in sub-block 408 may be in the form of a predicted time for the application 304 to finish rendering the first frame 302(1) or may be in the form of a predicted amount of time it will take the application 304 to render the first frame 302(1) (e.g., from start to finish). Thus, sub-blocks 406 and 408 describe how the prediction of the application rendering time 312(1) for the first frame 302(1) is based on one or more past rendering times 312 of a previous frame 302 (i.e., a frame 302 that came before the first frame 302(1)). However, it should be understood that the prediction of the application rendering time 312(1) may be based on a variety of other factors, such as scene complexity, current processing load, etc.

[0057] Simply put, application rendering time 312(1) (i.e., the amount of time for application 304 to render first frame 302(1)) is variable, but scan-out time 316(1) for first frame 302(1) is fixed, and scan-out time 316(1) plus illumination time 314(1) may correspond to the fastest frame rate in the range of possible frame rates. Thus, if the frame rate is pushed to, say, 144 Hz at the high end for a variable refresh rate display, pixel data is scanned out and an image is presented on the display for each frame 302 approximately every 6.9 ms. In other words, it should take approximately 6.9 ms to scan out pixel data from the frame buffer and illuminate the pixels on the display for a given frame 302 (for a display system that can achieve a refresh rate of 144 Hz at the high end). The illumination time 314(1), which occurs (technically is a period of time) after the pixel data is scanned out, is very short compared to both the application rendering time 312(1) and the scan-out time 316(1). The illumination time 314(1) also includes a settling time. For example, in an LCD, the crystals often take a few extra ms to settle, so the light-emitting elements 106 / 206 are illuminated after the crystals have settled. Some or all of these periods may be taken into account to determine the predicted illumination time 314(1) in sub-block 404. At a minimum, because the frame rate is variable, the predicted illumination time 314(1) is determined based at least in part on the predicted application rendering time 312(1) for the first frame 302(1). In some embodiments, such as in a rolling band type display driving scheme, the prediction of the illumination time in sub-block 404 may be based on a prediction of where the user 102 / 202 will look on the display panel 104 / 204, which may be determined from eye tracking data generated by an eye tracking system of the HMD 100 / 200.

[0058] In some embodiments, in sub-block 406, the logic may determine application rendering times 312 for multiple previous frames 302 and may determine an average application rendering time 312 based on these past rendering times to predict application rendering time 312(1) for a first frame 302(1). For example, the logic may determine a first amount of time the application 304 spent rendering the previous frame 302 and a second amount of time the application 304 spent rendering a previous frame 302 that precedes the previous frame 302, and then an average rendering time 312 may be determined based on the first amount of time and the second amount of time. The number of previous frames 302 to consider for this running average rendering time calculation is configurable (e.g., calculating the average rendering time 312 based on the rendering times 312 of the 10 most recently rendered (previous) frames 302). Taking such an average may provide the advantage of not reacting to anomalous rendering times, but instead waiting for rendering times to settle down a bit before adjusting the lighting time prediction.

[0059] In sub-block 410, logic (e.g., composite 308) may determine a predicted pose that HMD 100 / 200 will have at the predicted illumination time 314(1) determined in sub-block 404. This is the original prediction of the pose of HMD 100 / 200 for the first frame 302(1) and may be based at least in part on head tracking data generated by a head tracking system of HMD 100 / 200 (or HMD system).

[0060] In sub-block 412, logic (eg, compositor 308) may send pose data to application 304 indicating a predicted pose in sub-block 410 for rendering first frame 302(1).

[0061] In sub-block 414, logic (e.g., compositor 308) may receive pixel data for the first frame 302(1) from the application 304. The pixel data may include pixel values ​​for individual pixels in an array of pixels of the display panel 104 / 204, as described herein.

[0062] In sub-block 416, the logic (e.g., the composite 308) may determine a new predicted illumination time 314(1) representing the time the light elements 106 / 206 will illuminate for the first frame 302(1) based at least in part on the amount of time the application 304 spent rendering the first frame 302(1). The actual rendering time 312 of the first frame 302(1) (which may be determined in sub-block 418) may be less than the original predicted rendering time 312, in which case the composite 308 receives pixel data from the application 304 sooner than predicted. The actual rendering time 312 of the first frame 302(1) may be greater than the original predicted rendering time 312, in which case the composite 308 receives pixel data from the application 304 later than predicted. While the predicted application rendering time 312 may have been accurate, meaning that the pixel data was received exactly when it was expected to be received, there may often be a delta, albeit a small one, between the predicted rendering time and the actual rendering time for the first frame 302(1).

[0063] In sub-block 420, logic (e.g., composite 308) may determine a new predicted pose that HMD 100 / 200 will assume at a new predicted illumination time 314(1) based at least in part on head tracking data generated by the head tracking system of HMD 100 / 200. The delta between the prediction of the original pose and the prediction of the new pose depends on the accuracy of the prediction of application rendering time 312. However, as shown, this delta is likely to be small given the relatively short duration of playtime.

[0064] In sub-block 422, logic (e.g., compositor 308) may apply a reprojection adjustment to the pixel data for the first frame 302(1) to obtain modified (first) pixel data associated with the first frame 302(1). In the diagram of FIG. 3, this may occur after receiving pixel data for the first frame 302(1) from application 304 during rendering workload 306(1) of compositor 308. The reprojection adjustment may be determined based at least in part on the original predicted pose (determined in sub-block 410) and the new predicted pose (determined in sub-block 420), such as by comparing the two sets of pose data to determine an offset. In an illustrated example, compositor 308 may receive pixel data for the first frame 302(1) 1 or 2 ms earlier than predicted, or 1 or 2 ms later than predicted. Because the pixel data is output to a frame buffer and scanned out to the display as fast as possible, the updated prediction of the illumination time 314(1) may differ slightly from the original predicted illumination time 314(1) (e.g., by as little as 1 or 2 ms), and the HMD pose prediction may be updated based on this small delta to determine the reprojection adjustments to be made to the pixel data.

[0065] In sub-block 424, logic (e.g., compositor 308) may output modified (first) pixel data for the first frame 302(1) to a frame buffer. Again, in the case of an HMD 100 / 200 having a pair of display panels 104 / 204, this pixel data may correspond to a frame representing a pair of images displayed on the pair of display panels 104 / 204. As indicated by off-page reference "A," process 400 may continue to the first block of process 500 shown in FIG. 5.

[0066] 5 illustrates a flow diagram of an exemplary process 500 for dynamically determining values ​​of light output parameters for illuminating a low-persistence variable refresh rate display during presentation of an image, according to an embodiment disclosed herein. As indicated by off-page reference "A" in FIGS. 4 and 5, process 500 may continue from process 400. Process 500 may also be performed independently based on any pixel data output to a frame buffer. For discussion purposes, process 500 will be described with reference to the previous figure.

[0067] At 502, logic of a display system (e.g., a display controller, a display driver circuit, etc.) such as HMD 100 / 200 (or an HMD system) may cause an image to be presented on a display based at least in part on pixel data for frame 302 rendered, possibly by reprojection, by application 304, as described in process 400 of Figure 4. For discussion purposes, and following the example discussed in Figure 4, the image will be referred to as a first image corresponding to first pixel data of first frame 302(1). As indicated by sub-blocks 504 and 506, the presentation of the image in block 502 may include one or more sub-operations.

[0068] In sub-block 504, for example, the modified (first) pixel data for the first frame 302(1) may be scanned out onto the display panel 104 / 204 of the display system. For example, pixel values ​​for individual rows of pixels may be scanned out sequentially to the display panel 104 / 204 via a display port (e.g., HDMI), starting with a first subset (e.g., row) of pixels and ending with a last subset (e.g., row) of pixels.

[0069] In sub-block 506, the light emitting elements 106 / 206 of the display may be illuminated during the presentation of a first image on the display. The illumination may occur by emitting light pulses 108 / 208. That is, the low persistence display system may pulse the light emitting elements 106 / 206 on and off over a series of frames at a refresh rate of the display, which may be variable as described herein. As described herein, the pulsing of light during illumination in sub-block 506 may be a global flash of illumination where the light emitting elements 106 / 206 illuminate simultaneously, or a rolling band of illumination where individual subsets of the light emitting elements 106 / 206 illuminate sequentially during the light pulse 108 / 208. As illustrated by sub-blocks 508-514, values ​​of light output parameters for illuminating the light emitting elements 106 / 206 during the presentation of the first image may be determined such that the brightness of the display remains constant over a series of frames.

[0070] In sub-block 508, logic of the display system (e.g., HMD 100 / 200) may determine a first time difference 118(1) / 218(1) between a light pulse 108(0) (or illumination) of the light emitting element 106 / 206 for a previous frame 302 in the series of frames and an upcoming light pulse 108(1) (or illumination) of the light emitting element 106 / 206 for a first frame 302(1). The first time difference 118(1) / 218(1) may be referred to herein as a current frame time and may be derived from a current refresh rate of the display. In the illustrated embodiment, a first time difference 118(1) / 218(1) (or current frame time) of 8.3 ms is considered, which indicates an instantaneous refresh rate of 120 Hz. It should be appreciated that determining the first time difference 118(1) / 218(1) between a pair of consecutive light pulses 108 / 208 corresponding to a pair of consecutive frames 302 may include determining the time difference between any two corresponding points between the pair of light pulses 108 / 208 or the pair of frames. For example, the time difference 118 / 218 may be determined between the start of each light pulse 108 / 208, the midpoint of each light pulse 108 / 208, the end of each light pulse 108 / 208, etc. The display system logic, in some embodiments, may include frame start markers in the pixel data between frames rendered by each application, and these frame start markers may be used to determine the first time difference 118(1) / 218(1) in sub-block 508.

[0071] In sub-block 510, logic of the display system (e.g., HMD 100 / 200) may determine a first value of a light output parameter based at least in part on the first time difference 118(1) / 218(1). As described herein, the value of the light output parameter may correspond to the magnitude of the light pulse 108(1) for the first frame 302(1). Another value of the light output parameter may correspond to the duration of the light pulse 108(1) for the first frame 302(1). In a rolling band type display driving scheme, the value of the light output parameter may correspond to a number of light emitting elements 106 / 206 that are illuminated simultaneously during the light pulse 108(1) (e.g., the thickness of a rolling band of illumination). In some embodiments, more than one of these light output parameters may be determined and applied in combination to control the light output.

[0072] As part of determining the first value of the light output parameter in sub-block 512, the logic may determine a ratio between the first time difference 118(1) / 218(1) and the reference frame time 112 / 212. As described herein, the reference frame time 112 / 212 may be a time corresponding to an intermediate frame rate between a minimum frame rate and a maximum frame rate of a range of frame rates targeted by the application 304. For example, if the application 304 targets a range of frame rates from 45 FPS to 144 FPS, the reference frame time used to determine the aforementioned ratio may correspond to a frame time at a frame rate of 90 FPS, which is approximately a frame time of 11.11 ms. In this example, the reference frame time 112 / 212 would be 11.11 ms. However, this is merely one example, and the reference frame time 112 / 212 is configurable based in part on the specifications of the display system in which the techniques described herein are implemented. In the illustrated example, if first time difference 118(1) / 218(1) (or current frame time) is 8.3 ms and reference frame time 112 / 212 is 11.11 ms, then the ratio determined in sub-block 512 would be approximately 0.75. This can be thought of as a light output factor that is 75% of the nominal light output at the reference frame rate 112 / 212, meaning that the light output is reduced in some way (whether it be a reduction in the magnitude or duration of light pulse 108(1) or the like). In either case, a first value of the light output parameter may be determined based at least in part on this ratio.

[0073] In sub-block 514, the light emitting elements 106 / 206 may be controlled to emit light pulses 108(1) (or illumination) according to a first value of the light output parameter during presentation of a first image on the display. This process 500 may be repeated over a series of frames by dynamically determining a value of the light output parameter for each frame 302 of the series of frames. As a result of varying the light output parameter over the series of frames 302 in this manner, the brightness of the display remains substantially constant over the series of frames, thereby eliminating or at least reducing flickering of the display.

[0074] FIG. 6 illustrates exemplary components of an HMD 600 (or HMD system including HMD 600), such as a VR headset, according to embodiments disclosed herein that may be embodied. HMD 600 may be the same as or similar to HMD 100 / 200 referenced in the previous figures, and thus the components of HMD 600 illustrated in FIG. 6 may be implemented in HMD 100 / 200. HMD 600 may be implemented as a standalone device to be worn by user 102 (e.g., on the head of user 102). In some embodiments, HMD 600 may be head-mountable, such as by allowing user 102 to fasten HMD 600 on the user's head using a fastening mechanism (e.g., an adjustable band) sized to fit around the head of user 102. In some embodiments, HMD 600 includes a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display. Thus, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", and "head mounted display (HMD)" may be used interchangeably herein to refer to the device 600 of FIG. 6. However, it should be understood that these types of devices are merely examples of the HMD 600, and it should be understood that the HMD 600 may be implemented in a variety of other form factors. It should also be understood that some or all of the components shown in FIG. 6 may be implemented on the HMD 600. Thus, in some embodiments, a subset of the components shown in FIG. 6 may be implemented on a computing device that is part of the HMD system but separate from the HMD 600 itself, such as a PC, a game console, or any other suitable computing device. It should also be understood that non-HMD systems may include some similar components to implement at least the techniques and processes of dynamic light output adjustment described herein.

[0075] In the described implementation, the HMD 600 includes one or more processors 602 and memory 604 (e.g., computer-readable media 604). In some implementations, the processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU) 603, both a CPU and a GPU 603, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively, or in addition, what is described functionally herein may be implemented at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like. In addition, each of the processors 602 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0076] Memory 604 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Memory 604 may be implemented as a computer-readable storage medium ("CRSM"), which may be any available physical medium accessible by processor 602 to execute instructions stored in memory 602. In one basic implementation, CRSM may include random access memory ("RAM") and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store the desired information and that can be accessed by the processor 602.

[0077] In general, the HMD 600 (or HMD system) may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 604 is shown as including various modules, such as instructions, data stores, etc., which may be configured to execute on the processor 602 to perform the techniques, functions, and / or operations described herein. Although several exemplary functional modules are shown as stored on the computer-readable medium 604 and executable on the processor 602, the same functions may alternatively be implemented in hardware, firmware, or as a system-on-a-chip (SOC) and / or other logic.

[0078] The operating system module 606 may be configured to manage hardware within the HMD 600 and coupled to the HMD 600 for the convenience of other modules. Additionally, in some instances, the HMD 600 may include one or more applications 304 stored in the memory 604 or otherwise accessible to the HMD 600. In this implementation, the application 304 includes a game application 608. However, the HMD 600 may include any number or type of applications and is not limited to the particular example shown. The game application 608 may be configured to initiate gameplay of a video-based interactive game (e.g., a VR game) playable by the user 102 and output frames 302 to be rendered on a display panel of the HMD 600. The compositor 308, in combination with other logic of the HMD 600, may be configured to perform the techniques described herein to render the frames 302 by reprojection, such as by implementing the process 400. The light output adjustment component 610, in combination with other logic of the HMD 600, may be configured to implement the techniques described herein to dynamically adjust the light output of the light emitting elements 106 / 206 based on the time difference between successive light pulses 108 / 208 corresponding to successive frames 302.

[0079] Generally, the HMD 500 has input devices 612 and output devices 614. The input devices 612 may include control buttons. In some implementations, one or more microphones may function as input devices 612 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) may function as input devices 612 to receive gestural input, such as hand and / or head movements of the user 102. In some embodiments, additional input devices 612 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, handheld controller, etc. In other embodiments, the HMD 600 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 600 may be implemented with a relatively simple form of input device 612, a network interface (wireless or wired based), a power source, and processing / memory capabilities. For example, a limited set of one or more input components (e.g., dedicated buttons for initiating configuration, powering on / off, etc.) may be used and the HMD 600 may thereafter be used. In one implementation, the input device 612 may include controls such as basic volume control buttons for increasing / decreasing the volume, as well as power and reset buttons.

[0080] The output device 614 may include a display 616, which may include one or more display panels 104 / 204 (e.g., a stereo pair of display panels 104 / 204) including light emitting elements 106 / 206 as described herein. The output device 614 may further include, but is not limited to, light elements (e.g., LEDs), vibrators for creating tactile sensations, speakers (e.g., headphones), and the like. There may also be simple light elements (e.g., LEDs) to indicate a state, such as when the power is on.

[0081] The HMD 600 may further include a wireless unit 618 coupled to an antenna 620 to facilitate wireless connection to a network. The wireless unit 618 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the HMD 600 may further include a physical port to communicate with a network, connected peripheral devices (including PCs, game consoles, etc.), or other wireless networks, and to facilitate a wired connection to a plug-in network device that may be part of the HMD system.

[0082] The HMD 600 may further include an optical subsystem 622 that directs light from the electronic display 616 to the user's eyes using one or more optical elements. The optical subsystem 622 may include various types and combinations of different optical elements, including, but not limited to, apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 622 may have one or more coatings, such as an anti-reflective coating. The magnification of the image light by the optical subsystem 622 may allow the electronic display 616 to be physically smaller, lighter, and consume less power than larger displays. In addition, the magnification of the image light may increase the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using nearly all (e.g., 120-150 degrees diagonal), and possibly all, of the user's FOV. AR applications may have a narrower FOV (e.g., FOV of about 40 degrees). The Optical Subsystem 622 may be designed to correct one or more optical errors such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, content provided to the Electronic Display 616 for display is pre-distorted, and the Optical Subsystem 622 corrects the distortion upon receiving content-based generated image light from the Electronic Display 616.

[0083] The HMD 600 may further include one or more sensors 624, such as sensors used to generate motion, position, and orientation data. These sensors 624 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors 624 may also include secondary portions of sensors, such as a set of active or passive markers that may be viewed externally by a camera or color sensor to generate motion, position, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared or visible lights), that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more reference points for interpretation by software to generate motion, position, and orientation data. The HMD 600 may include a light sensor that is sensitive to light (e.g., infrared or visible light) projected or scattered by a base station in the environment of the HMD 600.

[0084] In one embodiment, the sensor 624 may include an inertial measurement unit (IMU) 626. The IMU 626 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU 626, or some combination thereof. Based on the measurement signals, such motion-based sensors, such as the IMU 626, may generate calibration data indicative of an estimated position of the HMD 600 relative to an initial position of the HMD 600. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). The IMU 626 may, for example, rapidly sample the measurement signals and calculate an estimated position of the HMD 600 from the sampled data. For example, the IMU 626 may integrate measurement signals received from an accelerometer over time to estimate a velocity vector, and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 600. A reference point is a point that may be used to describe the position of the HMD 600. A reference point may generally be defined as a point in space, but in various embodiments, a reference point is defined as a point within the HMD 600 (e.g., the center of the IMU 626). Alternatively, the IMU 626 provides sampled measurement signals to an external console (or other computing device), which determines the calibration data.

[0085] The sensor 624 may operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data may be generated at a rate of 1000 Hz (or one sensor reading every millisecond). In this manner, 1000 readings are generated per second. If the sensor generates this much data at this rate (or even faster), the data set used to predict movement becomes very large, even over a relatively short period of time, such as tens of milliseconds.

[0086] As mentioned, in some embodiments, the sensor 624 may include a light sensor that is sensitive to light emitted by a base station in the environment of the HMD 600 for purposes of tracking the position and / or orientation, posture, etc. of the HMD 600 in 3D space. Calculations of position and / or orientation may be based on timing characteristics of the light pulses and the presence or absence of light detected by the sensor 624.

[0087] The HMD 600 may further include an eye tracking system 628 that generates eye tracking data. The eye tracking system 628 may include, but is not limited to, a camera or other optical sensor internal to the HMD 600 to capture image data (or information) of the user's eyes, and the eye tracking system 628 can use the captured data / information to determine the motion vectors, interpupillary distance, interocular distance, three-dimensional (3D) position of each eye relative to the HMD 600, including the magnitude of twist and rotation (i.e., roll, pitch, and yaw) and the direction of gaze of each eye. In one embodiment, infrared light is emitted within the HMD 600 and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system 628 and analyzed to extract eye rotation from the change in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 102 may be used by the eye tracking system 628. Thus, the eye tracking system 628 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) per eye, and at least a subset of the tracked quantities may be combined from the two eyes of the user 102 to estimate a gaze point (i.e., a 3D place or location within the virtual scene where the user is looking) that may be mapped to a location on the display panel 104 / 204 to predict where the user 102 is looking with respect to an individual subset (e.g., a row) or a group of consecutive subsets (e.g., a group of consecutive rows) of pixels of the display panel 104 / 204. For example, the eye tracking system 628 may integrate information from past measurements, measurements identifying the position of the user 102's head, and 3D information describing the scene presented by the electronic display 616. Thus, the eye position and orientation information of the user 102 is used to determine the gaze point in the virtual scene presented by the HMD 600 at which the user 102 is looking, and to map that gaze point to a location on the display panel 104 / 204 of the HMD 600. The motion vectors may be utilized to predict eye movement trajectories, etc.

[0088] The HMD 600 may further include a head tracking system 630. The head tracking system 630 may utilize one or more of the sensors 624 to track head movements, including head rotation, of the user 102, as described above. For example, the head tracking system 630 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of the HMD 600. These calculations may be performed for every frame 302 in the sequence of frames 302, allowing the application 304 to determine how to render the scene in the next frame 302 depending on the head position and orientation, and allowing the compositor 308 to perform computational processing of reprojection adjustments. In some embodiments, the head tracking system 630, and / or the compositor 308 using the head tracking system 630, are configured to predict the future pose (position and / or orientation) of the HMD 600 based on current and / or historical data and / or based on known / implied scan-out latencies of individual subsets of pixels in a display system implementing a rolling band display driving technique, as described herein. This is because the application 304 is required to render a frame 302 before the user 102 actually sees the light (and, therefore, the image) on the display 616. Thus, the next frame 302 may be rendered based on a future prediction of the head position and / or orientation made at an earlier point in time, which may be in the range of approximately 12-30 milliseconds (ms) prior to rendering the frame 302, depending on the instantaneous frame rate and / or refresh rate. The rotation data provided by the head tracking system 630 may be used to determine both the direction of rotation of the HMD 600 and the amount of rotation of the HMD 600, in any suitable units of measure. For example, the direction of rotation may be simply output in terms of positive or negative horizontal and positive or negative vertical, corresponding to left, right, up, and down. The amount of rotation may be expressed in degrees, radians, etc. Angular velocity may be calculated to determine the rate of rotation of the HMD 600.

[0089] Although the present subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as exemplary forms of implementing the claims. The inventions described in the claims at the time of filing of this application are set forth below. [1] A display system comprising: a display having an array of light emitting elements, the display being configured to support a variable refresh rate; one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the display system to: outputting first pixel data associated with a first frame of the sequence of frames to a frame buffer; determining a first time difference between illumination of the light emitting element for a previous frame of the series of frames and a next illumination of the light emitting element for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and illuminating the light emitting elements according to the first value of the light output parameter during presentation of the first image on the display. [2] The computer-executable instructions, when executed by the one or more processors, cause the display system to: outputting second pixel data associated with a second frame of the series of frames to the frame buffer; and determining a second time difference between an illumination of the light emitting element for the first frame and a subsequent illumination of the light emitting element for the second frame; determining a second value of the light output parameter based at least in part on the second time difference; and presenting a second image on the display based at least in part on the second pixel data; and and illuminating the light emitting elements according to the second value of the light output parameter during presentation of the second image on the display. the second time difference is greater than the first time difference; the second value of the light output parameter causes the light emitting element to emit light at a greater brightness or for a longer duration than the first value of the light output parameter; the second time difference is less than the first time difference; The display system of [1], wherein the second value of the light output parameter causes the light-emitting element to emit light at a brightness less than or for a duration shorter than the first value of the light output parameter. [3] The first value of the light output parameter is the magnitude of a pulse of light for illuminating said light-emitting element; the duration of a pulse of the light to illuminate the light-emitting element; or The display system of [1], wherein the number of light-emitting elements illuminated simultaneously while sequentially illuminating individual subsets of the light-emitting elements during the presentation of the first image on the display corresponds to at least one of the number of light-emitting elements illuminated simultaneously while sequentially illuminating individual subsets of the light-emitting elements. determining the first value of the light output parameter based at least in part on the first time difference; determining a ratio of the first time difference and a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio. [5] The display system of [4], wherein the application that rendered the first frame is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, and the reference frame time corresponds to a frame time corresponding to an intermediate frame rate between the minimum frame rate and the maximum frame rate. [6] The display is a head mounted display (HMD), and the display system further comprises a head tracking system, and the computer-executable instructions, when executed by the one or more processors, prior to outputting the first pixel data to the frame buffer, provide the display system with: determining a predicted illumination time representative of a time that the light emitting element will illuminate for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the time that the light emitting element will illuminate for the first frame based at least in part on an amount of time the application spent rendering the first frame; and determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; and applying a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame. [7] determining the predicted illumination time, determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The display system of [6], wherein the predicted illumination time is determined based at least in part on the predicted rendering time. [8] The computer-executable instructions, when executed by the one or more processors, cause the display system to: determining a second amount of time that the application spent rendering a previous frame that precedes the previous frame; determining an average rendering time based at least in part on the first amount of time and the second amount of time; The display system of [7], wherein the determining of the predicted rendering time is based at least in part on the average rendering time. [9] The display system described in [1], wherein the display system is at least one of a virtual reality (VR) display system or an augmented reality (AR) display system.

[10] A method implemented by a display system including a display having an array of light emitting elements, the method comprising: outputting first pixel data to a frame buffer, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between a light pulse of the light emitting element for a previous frame of the series of frames and a next light pulse of the light emitting element for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and controlling the light emitting elements to emit light pulses according to the first value of the light output parameter during presentation of the first image on the display.

[11] The first value of the light output parameter is the magnitude of the write pulse for the first frame; the duration of the write pulse for the first frame; or The method according to

[10] , wherein the number of light-emitting elements simultaneously illuminated during the light pulse corresponds to at least one of the number of light-emitting elements simultaneously illuminated during the light pulse.

[12] determining the first value of the light output parameter based at least in part on the first time difference, determining a ratio of the first time difference and a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

[13] The display is a head mounted display (HMD), and the method further comprises, before outputting the first pixel data to the frame buffer: determining an expected illumination time representative of a time when the light pulse will occur for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by a head tracking system of the display system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the time at which the light pulse for the first frame will occur based at least in part on an amount of time the application spent rendering the first frame; and determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system;

[10] . The method of claim 10, further comprising: applying a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame.

[14] determining a first amount of time that the application spent rendering the previous frame; and determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The method of any one of claims 13 to 18, wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

[15] The controlling of the light emitting elements to emit the light pulses, causing the light-emitting elements to emit light simultaneously; or

[10] The method of claim 10, further comprising at least one of: causing individual subsets of the light-emitting elements to sequentially emit the light.

[16] A system comprising: a display having an array of light sources; a display driver circuit coupled to the array of light sources; and one or more processors, the one or more processors: outputting first pixel data to a frame buffer, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between an illumination of the light source for a previous frame of the series of frames and a next illumination of the light source for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and causing the display driver circuit to illuminate the light source according to the first value of the light output parameter during presentation of the first image on the display.

[17] The first value of the light output parameter is the magnitude of a pulse of light for illuminating said light source; the duration of the light pulse to illuminate the light source; or The system of

[16] , wherein during the presentation of the first image on the display, a number of light sources are illuminated simultaneously while sequentially illuminating individual subsets of the light sources.

[18] Determining the first value of the light output parameter based at least in part on the first time difference, determining a ratio of the first time difference and a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

[19] The method further includes a head tracking system, the display being a head mounted display (HMD), and the one or more processors, before outputting the first pixel data to the frame buffer: determining a predicted illumination time representative of a time that the light source will illuminate for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time that represents the time that the light source will illuminate for the first frame based at least in part on an amount of time the application spent rendering the first frame; and determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; and applying a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame.

[20] wherein the one or more processors: determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; 20. The system of claim 19, wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

Claims

1. 1. A display system comprising: a display having an array of light emitting elements, the display being configured to support a variable refresh rate; one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the display system to: outputting first pixel data associated with a first frame of the series of frames; determining a first time difference between an illumination of the light-emitting element for a previous frame of the series of frames and a next illumination of the light-emitting element for the first frame, where the previous frame and the first frame are consecutive frames, and the first time difference is one of: (1) between a start point of the illumination and a start point of the next illumination; (2) between a midpoint of the illumination and a midpoint of the next illumination; or (3) between an end point of the illumination and an end point of the next illumination. determining a ratio of the first time difference to a reference frame time; determining a first value of a light output parameter based at least in part on the ratio; and presenting a first image on the display based at least in part on the first pixel data; and illuminating the light emitting elements in accordance with the first value of the light output parameter during presentation of the first image on the display.

2. The computer-executable instructions, when executed by the one or more processors, cause the display system to: outputting second pixel data associated with a second frame of the series of frames; determining a second time difference between an illumination of the light emitting element for the first frame and a subsequent illumination of the light emitting element for the second frame; determining a second value of the light output parameter based at least in part on the second time difference; and presenting a second image on the display based at least in part on the second pixel data; illuminating the light emitting elements according to the second value of the light output parameter during presentation of the second image on the display; the second time difference is greater than the first time difference; the second value of the light output parameter causes the light emitting element to emit light at a greater brightness or for a longer duration than the first value of the light output parameter; the second time difference is less than the first time difference; 2. The display system of claim 1, wherein the second value of the light output parameter causes the light emitting elements to emit light at a brightness less than or for a duration shorter than the first value of the light output parameter.

3. The first value of the light output parameter is: the magnitude of a light pulse for illuminating said light emitting element; the duration of the light pulse for illuminating the light-emitting element; or The display system of claim 1 , wherein the sequential illumination of individual subsets of the light-emitting elements during the presentation of the first image on the display corresponds to at least one of the number of the light-emitting elements that are simultaneously illuminated during the sequential illumination of individual subsets of the light-emitting elements.

4. 2. The display system of claim 1, wherein an application that rendered the first frame is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, and the reference frame time corresponds to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

5. and wherein the display is a head mounted display (HMD), the display system further comprises a head tracking system, and the computer executable instructions, when executed by the one or more processors, prior to outputting the first pixel data, provide the display system with: determining a predicted illumination time representing a period during which the light emitting elements will illuminate the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the period during which the light emitting elements will illuminate the first frame based at least in part on an amount of time the application spent rendering the first frame; determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; 2. The display system of claim 1, further comprising: applying a reprojection adjustment to the pixel data to obtain the first pixel data associated with the first frame based at least in part on a comparison of the predicted pose and the new predicted pose, and wherein applying the reprojection adjustment transforms the pixel data to compensate for inaccuracies in the predicted pose.

6. determining the predicted illumination time, determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The display system of claim 5 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

7. The computer-executable instructions, when executed by the one or more processors, cause the display system to: determining a second amount of time that the application spent rendering a previous frame that precedes the preceding frame; determining an average rendering time based at least in part on the first amount of time and the second amount of time; The display system of claim 6 , wherein said determining said predicted rendering time is based at least in part on said average rendering time.

8. The display system of claim 1 , wherein the display system is at least one of a virtual reality (VR) display system or an augmented reality (AR) display system.

9. 1. A method implemented by a display system including a display having an array of light emitting elements, the method comprising: outputting first pixel data, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between a light pulse of the light emitting element for a previous frame of the series of frames and a next light pulse of the light emitting element for the first frame, where the previous frame and the first frame are consecutive frames, and the first time difference is one of: (1) between a start point of the light pulse and a start point of the next light pulse; (2) between a midpoint of the light pulse and a midpoint of the next light pulse; or (3) between an end point of the light pulse and an end point of the next light pulse. determining a ratio of the first time difference to a reference frame time; determining a first value of a light output parameter based at least in part on the ratio; and presenting a first image on the display based at least in part on the first pixel data; controlling the light emitting elements to emit light pulses according to the first value of the light output parameter during presentation of the first image on the display.

10. The first value of the light output parameter is: the magnitude of the write pulse for the first frame; the duration of the write pulse for the first frame, or The method of claim 9 , wherein the number of light emitting elements simultaneously illuminated during the light pulse for the first frame corresponds to at least one of the number of light emitting elements simultaneously illuminated during the light pulse for the first frame.

11. the display is a head mounted display (HMD), and the method further comprises, before outputting the first pixel data, determining an expected illumination time representative of a period during which the light pulse for the first frame will occur; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by a head tracking system of the display system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the period during which the light pulse for the first frame will occur based at least in part on an amount of time the application spent rendering the first frame; determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; 10. The method of claim 9, further comprising: applying a reprojection adjustment to the pixel data based at least in part on a comparison of the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame, where applying the reprojection adjustment transforms the pixel data to compensate for inaccuracies in the predicted pose.

12. determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The method of claim 11 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

13. said controlling said light emitting elements to emit said light pulses; causing the light-emitting elements to emit light simultaneously; or The method of claim 9 , comprising at least one of: causing individual subsets of the light emitting elements to sequentially emit the light.

14. 1. A system comprising: a display having an array of light sources; a display driver circuit coupled to the array of light sources; one or more processors, the one or more processors: outputting first pixel data, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between an illumination of the light source for a previous frame of the series of frames and a next illumination of the light source for the first frame, where the previous frame and the first frame are consecutive frames, and the first time difference is one of: (1) between a start point of the illumination and a start point of the next illumination; (2) between a midpoint of the illumination and a midpoint of the next illumination; or (3) between an end point of the illumination and an end point of the next illumination. determining a ratio of the first time difference to a reference frame time; determining a first value of a light output parameter based at least in part on the ratio; and presenting a first image on the display based at least in part on the first pixel data; and causing the display driver circuit to illuminate the light source according to the first value of the light output parameter during presentation of the first image on the display.

15. The first value of the light output parameter is: a light pulse magnitude for illuminating said light source; the duration of the light pulse for illuminating the light source; or 15. The system of claim 14, wherein during the presentation of the first image on the display, sequential illumination of individual subsets of the light sources corresponds to at least one of a number of the light sources simultaneously illuminated.

16. The method further comprises the step of: providing a head tracking system, the display being a head mounted display (HMD), and the one or more processors, before outputting the first pixel data, determining an expected illumination time representing a period during which the light source will illuminate the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the period during which the light source will illuminate the first frame based at least in part on an amount of time the application spent rendering the first frame; determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; 15. The system of claim 14, further configured to apply a reprojection adjustment to the pixel data to obtain the first pixel data associated with the first frame based at least in part on a comparison of the predicted pose and the new predicted pose, where applying the reprojection adjustment transforms the pixel data to compensate for inaccuracies in the predicted pose.

17. the one or more processors: determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The system of claim 16 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

18. 15. The system of claim 14, wherein an application that rendered the first frame is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, and the reference frame time corresponds to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

19. The system of claim 14 , wherein the system is at least one of a virtual reality (VR) display system or an augmented reality (AR) display system.

20. 10. The method of claim 9, wherein an application that rendered the first frame is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, and the reference frame time corresponds to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

Images