Display system with dynamic light output adjustment for maintaining constant brightness

Abstract

The light output of a display can be dynamically adjusted on-the-fly. When implemented on a low visual persistence display that supports variable refresh rates, the adjustment maintains constant luminance over a series of frames to eliminate flicker of the display. When pixel data for a given frame is output to a frame buffer for rendering an image on the display, a time difference between illumination of a light-emitting element of the display for a previous frame and upcoming illumination of the light-emitting element for the given frame can be determined, and the time difference used to determine a value of a light output parameter. During rendering of the image, the light-emitting element can be illuminated according to the value of the light output parameter. The determination is iterated over a series of frames to dynamically adjust the light output of the display.

Description

[0001] Cross-references to related applications

[0002] This PCT application claims priority to U.S. Patent Application Serial No. 16 / 399,804, filed April 30, 2019, entitled “DISPLAY SYSTEM WITH DYNAMICLIGHT OUTPUT ADJUSTMENT FOR MAINTAINING CONSTANT BRIGHTNESS”, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Traditional displays used in virtual reality (VR) systems (such as those embedded in VR headsets) operate at a fixed refresh rate. A display's "refresh rate" is the number of times the display refreshes an image or redraws the screen per second (e.g., a 90 Hz display refreshes an image 90 times per second). Advances in graphics have spurred the development of displays that support variable refresh rates, meaning the display's refresh rate changes dynamically as frames are being rendered. This allows the refresh rate to be kept synchronized with the changing frame rates from graphics rendering applications such as video games. For example, G-SYNC... TM —A technology and FreeSync obtained from NVIDIA TM —A technology available from Advanced Micro Devices. Both provide logic for matching the display's refresh rate to the frame rate of a video game. These technologies eliminate screen tearing (e.g., where an image comprises some pixels from the previous frame and some from the current frame), and allow the video game being played to target a range of frame rates rather than a single frame rate, which, given the speed of the graphics processing unit (GPU), allows it to benefit from the fastest possible frame rate.

[0004] While variable refresh rate techniques are suitable for highly visual persistence displays, where the backlight is on for most of the time in a series of frames, these techniques are not suitable for VR systems, as most VR systems use low visual persistence displays that pulse the backlight on and off at the refresh rate of the display. If variable refresh rate is used with low visual persistence displays, the user viewing will notice flicker of the display as the luminance changes as well as the refresh rate changes. This is because, at higher refresh rates, the light pulses occur closer together in time, creating a brightening effect, while at lower refresh rates, the light pulses occur farther apart in time, creating a dimming effect. For a user wearing a head-mounted display (HMD), this manifests as a continuous brightening and dimming of the display (i.e., flicker). In effect, scenes with low complexity (e.g., with not many moving objects and simple textures) will appear very bright to the user, as the refresh rate can be higher for low complexity scenes, while scenes with high complexity (e.g., with many moving objects and complex textures) will appear very dim to the user, as the refresh rate can be lower for high complexity scenes. Thus, variable refresh rate display techniques are not yet suitable for VR systems that use low visual persistence displays.

[0005] Technical solutions are provided herein for improving and enhancing these and other systems. BRIEF DESCRIPTION OF DRAWINGS

[0006] The detailed description is described with reference to the accompanying figures. In these figures, the left-most digit(s) of the reference numbers identify the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0007] Figure 1 FIG. 1 is a schematic diagram illustrating an exemplary technique for dynamically adjusting the light output of a low visual persistence, variable refresh rate display to maintain constant luminance over a series of frames, in accordance with embodiments disclosed herein.

[0008] Figure 2 FIG. 1 is a schematic diagram illustrating an exemplary technique for dynamically adjusting the light output of a low visual persistence, variable refresh rate display to maintain constant luminance over a series of frames, in accordance with embodiments disclosed herein.

[0009] Figure 3 FIG. 1 is a schematic diagram illustrating an exemplary technique for dynamically adjusting the light output of a low visual persistence, variable refresh rate display to maintain constant luminance over a series of frames, in accordance with embodiments disclosed herein.

[0010] Figure 4A flowchart showing an exemplary process for rendering frames with re-projection in a low-visual-persistence, variable-refresh-rate display system, in accordance with embodiments disclosed herein, is shown.

[0011] Figure 5 A flowchart showing an exemplary process for dynamically determining values for light output parameters used to illuminate a low-visual-persistence, variable-refresh-rate display during image presentation, in accordance with embodiments disclosed herein, is shown.

[0012] Figure 6 Exemplary components of a system including a wearable device such as a VR headset in which the technology disclosed herein can be implemented are shown. DETAILED DESCRIPTION

[0013] Among other things, techniques and systems for dynamically adjusting light output of a low-visual-persistence, variable-refresh-rate display to maintain constant luminance over a series of frames are described herein. The display system can include a display having an array of light-emitting elements (or light sources). As mentioned, the display system can be a low-visual-persistence display system, meaning that the light-emitting elements emit light for a small fraction of the frame time. For example, for a given frame of a 90 Hz refresh rate, the light-emitting elements can illuminate (or emit a pulse of light) for a duration of about 1 ms out of a total frame time of about 11.11 ms. In an illustrative example, the display of the display system disclosed herein can be a head-mounted display (HMD) that can be worn by a user. In this example, the display system can be a virtual reality (VR) system or an augmented reality (AR) system. The display itself, such as the HMD, can include one or more display panels that present images based on frames output by a graphics rendering application, such as a video game. The user views these images through optics included in the HMD, causing the user to perceive these images as if the user is immersed in a VR or AR environment.

[0014] The displays described herein can support variable refresh rates. Thus, a graphics rendering application, such as a video game, can render frames at a variable frame rate, and the refresh rate of the display can be dynamically adjusted to match (or remain synchronized with) the frame rate of the application. In this way, the application can target a range of frame rates, and the refresh rate of the display can be associated with the frame rate of the application so that a single frame is rendered and a corresponding image is presented at each screen refresh. Because the refresh rate can dynamically vary, if the light output of the light-emitting elements remains constant for each pulse of light, the luminance of the display can fluctuate over a series of frames, and this can be perceived by the user as flicker of the display.

[0015] Accordingly, techniques and systems are described herein for maintaining a constant brightness of a low-persistence display that supports variable refresh rates. Logic of a disclosed display system can be configured to output first pixel data to a frame buffer for rendering a corresponding first image on a display, the first pixel data associated with a first frame of a series of frames. The logic can determine a first time difference between an illumination (or light pulse) of a light-emitting element of the display for a previous frame of the series of frames and an upcoming illumination (or light pulse) of the light-emitting element for the first frame corresponding to the first image to be rendered. Based on the first time difference, which is indicative of an instantaneous refresh rate of the display, the logic of the display system can determine a first value of a light output parameter. The light-emitting element can be controlled to illuminate according to the first value of the light output parameter during rendering of the first image on the display. The process can iterate over the series of frames by dynamically determining a value of the light output parameter for each frame, and as a result of varying the light output parameter in this manner over the series of frames, the display brightness remains substantially constant over the series of frames, thereby eliminating or at least mitigating flicker.

[0016] Generally, if the time difference between a pair of sequential light pulses corresponding to a pair of sequential frames is relatively short (meaning a relatively high refresh rate), the logic of the display system determines a value of the light output parameter that causes a reduction in the amplitude or duration of the light output from the light-emitting element. On the other hand, if the time difference between a pair of sequential light pulses is relatively long (meaning a relatively low refresh rate), the logic of the display system determines a value of the light output parameter that causes an increase in the amplitude or duration of the light output from the light-emitting element. In this manner, constant brightness is achieved by increasing the amplitude and / or duration of the light output at relatively low refresh rates and decreasing the amplitude and / or duration of the light output at relatively high refresh rates.

[0017] As mentioned, adjusting the light output of the light-emitting element of the display can be achieved in a variety of ways. For example, the light pulse for a given frame can be made higher in height than the previous frame or reference pulse (e.g., by controlling the light-emitting element to emit light at a higher luminous intensity). Conversely, the light pulse for a given frame can be made shorter in height than the previous frame or reference pulse (e.g., by illuminating the light-emitting element at a lower luminous intensity). Additionally or alternatively, to make the light pulse higher or shorter, the light pulse for a given frame can be made wider in width than the previous frame or reference pulse (e.g., by illuminating the light-emitting element for a longer duration). Conversely, the light pulse for a given frame can be made thinner in width than the previous frame or reference pulse (e.g., by illuminating the light-emitting element for a shorter duration). To maintain constant brightness, the area under the pulse (e.g., the area under a square waveform representing the pulse) can remain constant, despite the dynamic adjustment of the height and / or width of the pulse (or representative waveform thereof).

[0018] In some embodiments, a reference frame time is associated with a target brightness level that the display is to maintain constant. The reference frame time can be used in determining a value for a light output parameter. For example, in determining a time difference between sequential light pulses corresponding to a pair of sequential frames, the logic can determine a ratio of the time difference to the reference frame time, and the ratio can be used in determining a value for a light output parameter used to adjust light output of the light emitting elements for the current frame. The reference frame time can be a time corresponding to an intermediate frame rate between a minimum frame rate and a maximum frame rate in a frame rate range that an application targets. For example, if an application targets a frame rate range between 45 frames per second (FPS) and 144 FPS, the reference frame time used in determining the aforementioned ratio can correspond to a frame time at a frame rate of 90 FPS, which is a frame time of approximately 11.11 ms. This is merely an example, and the reference frame time is configured based in part on the specifications of the display system in which the techniques described herein are implemented.

[0019] Techniques and systems are also described herein for applying "re-projection" adjustments in low-visual-persistence, variable refresh rate display systems, such as display systems including HMDs. The described re-projection techniques can be used to compensate for slight inaccuracies in initial pose predictions of an HMD, which are in turn based on a predicted illumination time for a given frame that is predicted prior to rendering the given frame. Pixel data received for a given frame from an application can be modified using re-projection, such as by transforming (e.g., through rotation and re-projection calculations) the pixel data in a manner that takes into account updated predictions of the pose of the HMD after the actual rendering time of the frame is known to the display system. By presenting the correct pixels when light from the display reaches the user's eyes, undesirable visual artifacts can be mitigated using re-projection.

[0020] By dynamically adjusting light output based on a time difference between sequential light pulses of a pair of sequential frames, which indicates an instantaneous frame rate, flicker (i.e., fluctuations in brightness) of a display over a series of frames can be mitigated without eliminating the flicker of the display. This can be independent of re-projection, or can be combined with re-projection. When dynamic light output adjustment is combined with the re-projection techniques described herein, a more robust display system is provided that mitigates visual artifacts while also maintaining constant brightness of the display. In other words, the techniques and systems described herein allow for a display system that presents the correct scene that the user is currently looking at, and presents the correct scene while maintaining constant brightness of the display.

[0021] The techniques and processes disclosed herein can be implemented by a system such as an HMD system. Non-transitory computer-readable media storing computer-executable instructions for implementing the techniques and processes disclosed herein are also disclosed herein. Although the techniques and systems disclosed herein are discussed in the context of video game applications, particularly VR game applications, by way of example, it should be understood that the techniques and systems described herein can provide benefits for other applications, including but not limited to non-VR applications (e.g., AR applications) and / or non-game applications (such as industrial machine applications, defense applications, robotics applications, etc.). Moreover, the techniques and processes described herein can be implemented in non-HMD systems, such as other display systems that are not considered “near-eye” display systems or that do not involve wearable devices. Furthermore, although particular benefits are described for low-persistence, variable refresh rate display systems, the techniques and processes described herein can be implemented in other types of display systems, and it should be understood that the techniques and processes described herein are not necessarily limited to low-persistence display systems or variable refresh rate display systems.

[0022] Figure 1 is a schematic diagram illustrating an exemplary technique for dynamically adjusting the light output of a low-persistence, variable refresh rate display to maintain constant luminance over a series of frames, in accordance with the embodiments disclosed herein. Figure 1 A head-mounted display (HMD) 100 worn by a user 102 is depicted. The HMD 100 is an example of a display system that can implement the techniques and processes described herein. Thus, the HMD 100 can be referred to simply as a “display” or “display system” herein at times. Figure 1 The HMD 100 in the example of FIG. 1 can include a single display panel 104 or multiple display panels 104, such as left and right display panels in a stereoscopic display panel pair. The one or more display panels 104 in the HMD 100 can be used to present a series of image frames (referred to herein as “frames”) that can be viewed by the user 102 wearing the HMD 100. It should be understood that the HMD 100 can include any number of display panels 104 (e.g., more than two display panels, a pair of display panels, or a single display panel). Thus, the term “display panel” as used herein in the singular can refer to a single display panel 104 of a display system having any number of display panels (e.g., “display panel” can refer to either display panel 104 in a pair of display panels). In a dual-panel HMD 100, for example, a stereoscopic frame buffer can render, for example, 2160 x 1200 pixels (e.g., 1080 x 1200 pixels on each display panel) on both display panels of the HMD 100.

[0023] The HMD 100 can utilize any suitable type of display technology, such as an emissive display that utilizes light-emitting elements (e.g., light-emitting diodes (LEDs)) to emit light during the presentation of a frame on the display panel 104. As an example, the display panel 104 of the HMD 100 can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0024] The display panel 104 of the HMD 100 can support a variable refresh rate. Thus, the HMD 100 can operate within any suitable range of refresh rates, such as a range of 45 hertz (Hz) to 144 Hz, a range of 45 Hz to 120 Hz, a range of 90 Hz to 144 Hz, or any suitable range of refresh rates. The "refresh rate" of a display is the number of times per second that the display can refresh an image or redraw the screen. The number of frames displayed per second can be equivalent to the instantaneous refresh rate of the display. In other words, as a series of frames are processed (e.g., rendered) and as the images are presented on the display, the logic of the HMD 100 can target the presentation of an image for a single frame in the series of frames each time the screen is refreshed.

[0025] The HMD 100 can implement any suitable low-persistence driving scheme, including but not limited to a "global flash" type of display driving scheme or a "rolling band" type of display driving scheme. Using global flash technology, all of the light-emitting elements 106 (e.g., an M x N array of light-emitting elements 106, which are sometimes referred to herein as "light sources" 106) can be illuminated in synchrony to present a single image on the display for a given frame by illuminating all of the pixels of the display. Using rolling band technology, various subsets of light-emitting elements 106 and thus pixels can be independently and sequentially illuminated in the form of a rolling illuminated band during an illumination period. Such rolling band technology can be implemented by virtue of the light-emitting elements 106 being individually addressable. In some embodiments, both the pixel array and the light-emitting elements 106 on the display panel 104 are arranged in rows and columns, but not necessarily one pixel per light-emitting element 106. In such a configuration, for a rolling band type of display driving scheme, the light-emitting elements 106 of various rows and / or various columns can be addressed in sequence, and / or the light-emitting elements 106 of various groups of contiguous rows and / or various groups of contiguous columns can be addressed in sequence. As a result of addressing the light-emitting elements 106 in such a "rolling" manner, the subsets of pixels corresponding to these individually addressable subsets of light-emitting elements 106 can be independently "illuminated."

[0026] As used herein, “illuminating a pixel” means illuminating the light emitting element 106 corresponding to that pixel. For example, an LCD illuminates the light emitting element 106 with a backlight to illuminate the corresponding pixel of the display. Also, as used herein, a “subset of pixels” can include a single pixel or multiple pixels (e.g., a group of pixels). Likewise, a “subset of light emitting elements 106” can include a single light emitting element 106 or multiple light emitting elements 106 (e.g., a group of light emitting elements 106). In some embodiments, a subset of pixels includes a row of pixels, a column of pixels, a group of contiguous rows of pixels, or a group of contiguous columns of pixels. Likewise, a subset of light emitting elements 106 can include a row of light emitting elements 106, a column of light emitting elements, a group of contiguous rows of light emitting elements 106, or a group of contiguous columns of light emitting elements 106. Thus, in one aspect of the techniques and systems described herein, a subset of pixels can be sequentially (serially) scanned out and illuminated, such as by sequentially scanning out and illuminating each row of pixels starting with a first row of pixels (e.g., a top row of pixels) and ending with a last row of pixels (e.g., a bottom row of pixels), such as by illuminating corresponding subsets of light emitting elements in order. However, any suitable illumination pattern (e.g., a snake illumination pattern, column-by-column illumination, multiple rows / columns of pixels or light emitting elements 106 at a time in sequence, etc.) can be employed using the techniques and systems described herein.

[0027] To drive the display panel 104, among other things, the HMD 100 can include a display controller, such as a microcontroller or similar processor, a display driver circuit, and similar electronics for driving the display panel 104. The display driver circuit can be coupled to the array of light emitting elements 106 of the display panel 104 via conductive paths (such as metal traces) on a flexible printed circuit. In one example, the display controller can be communicatively coupled to the display driver circuit and configured to provide signals, information, and / or data to the display driver circuit. The signals, information, and / or data received by the display driver circuit can include one or more light output parameters that cause the display driver circuit to illuminate the light emitting elements 106 in a particular manner. That is, the display controller can determine which light emitting element(s) 106 to illuminate, when to illuminate the light emitting element(s) 106, the amplitude of the light output (e.g., the height of a pulse), and / or the duration of the light output (e.g., the width of a pulse), etc., and the display controller can communicate the appropriate signals, information, and / or data to the display driver circuit in order to achieve that purpose.

[0028] Pixel data for a given frame can be output to a frame buffer for rendering the frame as an image on the display panel of the HMD 100 or 104 thereof. In some embodiments, the pixel data for each frame can include a two-dimensional array of per-pixel values (e.g., color values). In some embodiments, the pixel data also includes additional data or metadata, such as depth values. In some embodiments, the pixel data can include data for each pixel represented by a single set of color values and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels). This pixel data can be output to a frame buffer (e.g., a stereoscopic frame buffer) to render the desired visual effect as an image on the display panel 104 of the HMD 100.

[0029] The HMD 100 can represent a VR headset for use in a VR system, such as for use with a VR gaming system. However, the HMD 100 can additionally or alternatively be implemented as an AR headset for use in AR applications. In AR, the user 102 sees virtual objects overlaid on a real-world environment, whereas in VR, the user 102 does not see the real-world environment and is fully immersed in a virtual environment as perceived via the display panel 104 and optics (e.g., lenses) of the HMD 100. The examples described herein primarily relate to a VR-based HMD 100, but it should be understood that the HMD 100 is not limited to implementation in VR applications, and the display system is not limited to implementation in the HMD 100.

[0030] Generally, a graphics-based application (e.g., a video game) executing on a computing device, such as the HMD 100 itself or a computing device (e.g., a personal computer (PC), a game console, etc.) associated with and coupled to the HMD 100 as part of a display / HMD system, can be configured to output a series of frames. The series of frames are ultimately rendered as images on the display panel 104 of the HMD 100. During the rendering of a single image corresponding to a single frame, the low persistence display system is configured to pulse the light emitting elements 106 of the display panel 104 such that light is emitted toward the eyes of the user 102, causing the user 102 to see the image rendered on the display during the light pulses 108. Figure 1The timeline 110 shown in FIG. 1 illustrates a series of light pulses 108(1), 108(2),... 108(N) (collectively, 108) that can occur at a variable refresh rate of a display system. Each of these light pulses 108 represents a discrete emission of light from the light-emitting element 106 for a given frame. Due to the logic of matching the refresh rate of the HMD 100 to the frame rate of the application that is rendering the frames, this series of light pulses 108 is offset from, but synchronized with, the rendering of the series of frames by the application.

[0031] As Figure 1 The HMD 100 transitions over time from a relatively higher refresh rate (towards the left side of the timeline 110) to a relatively lower refresh rate (towards the right side of the timeline 110), as shown in the timeline 110 in FIG. 1. In the illustrative example, the refresh rate towards the left side of the timeline 110 can be on the order of 144 Hz, while the refresh rate towards the right side of the timeline 110 can be on the order of 60 Hz. In this regard, Figure 1 The timeline 110 in FIG. 1 is not drawn to scale, but merely illustrates generally the variable refresh rate.

[0032] Figure 1 A reference frame time 112 (sometimes referred to as a "nominal" frame time) is shown, which can be associated with a target brightness level that the display is to maintain constant. Consider the following example: where the reference frame time 112 is approximately 11.11 ms, which corresponds to a refresh rate of 90 Hz. In this example, if the refresh rate were to remain constant at 90 Hz, the light-emitting element 106 would be uniformly illuminated over the series of frames according to the values of the light output parameters corresponding to the light pulses 108 having a pulse duration 114 and a pulse amplitude 116, as shown in the middle of the timeline 110 in FIG. 1. At the refresh rate corresponding to the reference frame time 112, the light pulses 108 having the pulse duration 114 and the pulse amplitude 116 achieve the target brightness level that the display is to maintain constant. Thus, as the refresh rate changes, light output adjustments are made dynamically in order to maintain the brightness constant at this target level. Figure 1 At the refresh rate corresponding to the reference frame time 112, the light pulses 108 having the pulse duration 114 and the pulse amplitude 116 achieve the target brightness level that the display is to maintain constant. Thus, as the refresh rate changes, light output adjustments are made dynamically in order to maintain the brightness constant at this target level.

[0033] During the relatively higher refresh rate towards the left side of the timeline 110 and at a time between the light pulses 108(1) and 108(2), the logic of the display system can determine a first time difference 118(1) between the light pulse 108(1) (i.e., the illumination of the light-emitting element 106 for a previous frame in the series of frames) and the upcoming light pulse 108(2) (i.e., the illumination of the light-emitting element 106 for a current frame). Based at least in part on the first time difference 118(1) (which is indicative of the instantaneous refresh rate), the logic can determine the values of the light output parameters. In Figure 1In the example, the light output parameter corresponds to or indicates the height of the light pulse 108(2) for the current frame. Therefore, this value of the light output parameter, determined based on the time difference 118(1), can be used to control the light-emitting element 106 to emit the light pulse 108(2) at a specific amplitude. This specific amplitude of the light pulse 108(2) is... Figure 1 The pulse amplitude 116 is shown as smaller than the target brightness level associated with reference frame time 112. In other words, because the refresh rate toward the left of timeline 110 is higher than the refresh rate corresponding to reference frame time 112 in the middle of timeline 110, the height or amplitude of the light pulse 108(2) can be adjusted to be smaller than the pulse amplitude 116 associated with reference frame time 112 via light output adjustment 120(1). That is, the dashed outline represents the “reference pulse” at pulse amplitude 116 and pulse duration 114, and the solid outline represents the actual light pulse 108(2), the reference pulse shown being used to illustrate the amplitude 116 of the light pulse 108(2) relative to the reference pulse. Controlling the luminous intensity of the light output for light pulse 108(2) may include driving the light-emitting element 106 with a lower current and / or voltage than the current and / or voltage that would be used to drive the light-emitting element 106 at a refresh rate corresponding to reference frame time 112. In some implementations, the control of the light output (or light output adjustment 120(1)) is performed digitally by setting a digital value to control the light output of the light pulse 108(2).

[0034] During a relatively low refresh rate toward the right of timeline 110 and in the time between light pulses 108(N-1) and 108(N), the logic of the display system can determine the Nth time difference 118(N) between the illumination of the previous frame by light pulse 108(N-1) or light-emitting element 106 for the current frame and the upcoming light pulse 108(N) or light-emitting element 106 for the current frame. Based at least in part on the Nth time difference 118(N) (which indicates the instantaneous refresh rate), the logic can determine the value of a light output parameter that corresponds to or indicates the height of light pulse 108(N) for the current frame. Therefore, this value of the light output parameter determined based on the time difference 118(N) can be used to control the light-emitting element 106 to emit light pulse 108(N) at a specific amplitude. This specific amplitude of light pulse 108(N) in… Figure 1The middle is shown to be greater than the pulse amplitude 116 associated with the target brightness level for the reference frame time 112. In other words, because the refresh rate toward the right side of the timeline 110 is lower than the refresh rate corresponding to the reference frame time 112 in the middle of the timeline 110, the height or amplitude of the light pulse 108(N) can be adjusted to an amplitude that is greater than the pulse amplitude 116 associated with the reference frame time 112 via the light output adjustment 120(N). Controlling the luminous intensity of the light output for the light pulse 108(N) can include driving the light emitting element 106 at a higher current and / or voltage than the current and / or voltage that would be used to drive the light emitting element 106 at the refresh rate corresponding to the reference frame time 112. In some embodiments, the control of the light output (or light output adjustment 120(N)) is performed digitally by setting a digitized value to control the light output of the light pulse 108(N). Notably, the display driving circuit has sufficient (or extra) bandwidth to provide the ability to increase the pulse height.

[0035] In some embodiments, the reference frame time 112 is used to determine a value for the light output parameter corresponding to Figure 1 The value of the amplitude of the light pulse 108 shown. For example, the logic of the display system can determine the ratio of the first time difference 118(1) to the reference frame time 112, and this ratio can be used to determine a value for the light output parameter for the light output adjustment 120(1) associated with the light pulse 108(2). Consider the following example: where the first time difference 118(1) is approximately 8.3 ms, which corresponds to a refresh rate of 120 Hz, and where the reference frame time 112 is approximately 11.11 ms, which corresponds to a refresh rate of 90 Hz. In this example, the following ratio can be calculated: In this example, the light pulse 108(2) can be adjusted to an amplitude that is approximately 75% of the pulse amplitude 116 associated with the target brightness level for the reference frame time 112. Because the pulses 108(1) and 108(2) are relatively close together in time, the downward adjustment of the pulse amplitude compensates for the increased brightness caused by the more closely spaced light pulses 108 at the relatively higher refresh rate, which maintains the constant brightness of the HMD 100.

[0036] Consider the following example: where the Nth time difference 118(N) is approximately 16.66 ms, which corresponds to a refresh rate of 60 Hz, and the reference frame time 112 is still approximately 11.11 ms. In this example, the following ratio can be calculated: In this example, the light pulse 108(N) can be adjusted to an amplitude of approximately 150% of the pulse amplitude 116 associated with the target luminance level for the reference frame time 112. Because the pulses 108(N-1) and 108(N) are far apart in time, the upward adjustment of the pulse amplitude compensates for the decrease in luminance caused by the more spread out light pulse 108 at the relatively lower refresh rate, which maintains constant luminance for the HMD 100.

[0037] Figure 2 is a diagram illustrating another example technique for dynamically adjusting light output of a low-visual-persistence, variable refresh rate display to maintain constant luminance over a series of frames in accordance with embodiments disclosed herein. As Figure 2 illustrated in the timeline 210 in Figure 1 , similar to the example of Figure 2 , the HMD 200 transitions over time from a relatively higher refresh rate (towards the left side of the timeline 210) to a relatively lower refresh rate (towards the right side of the timeline 210). In the illustrative example, the refresh rate towards the left side of the timeline 210 can be on the order of 144 Hz, while the refresh rate towards the right side of the timeline 210 can be on the order of 60 Hz. In this regard, Figure 2 the timeline 210 in

[0038] Figure 2 illustrates a reference frame time 212, which can also be associated with a target luminance level that the display is to maintain constant. At the refresh rate corresponding to the reference frame time 212, a light pulse 208 having a pulse duration 214 and a pulse amplitude 216 achieves the target luminance level that the display is to maintain constant. Thus, as the refresh rate changes, light output adjustments are dynamically made in order to maintain luminance constant at this target level.

[0039] During the relatively higher refresh rate towards the left side of the timeline 210 and at a time between the light pulses 208(1) and 208(2), logic of the display system can determine a first time difference 218(1) between the light pulse 208(1) (i.e., the illumination of the light emitting element 206 for a previous frame in the series of frames) and the upcoming light pulse 208(2) (i.e., the illumination of the light emitting element 206 for a current frame). Based at least in part on the first time difference 218(1) (which is indicative of the instantaneous refresh rate), the logic can determine a value of a light output parameter. In Figure 2 the example of Figure 2 , the light output parameter corresponds to or is indicative of a width of the light pulse 208(2) for the current frame. Thus, this value of the light output parameter determined based on the time difference 218(1) can be used to control the light emitting element 206 to emit the light pulse 208(2) for a particular duration. This particular duration of the light pulse 208(2) is in accordance with the value of the light output parameter determined based on the time difference 218(1).The width of the light pulse 208(2) is shown to be shorter than the pulse amplitude 214 associated with the target luminance level for the reference frame time 212. In other words, because the refresh rate toward the left side of the timeline 210 is higher than the refresh rate corresponding to the reference frame time 212 in the middle of the timeline 210, the width or duration of the light pulse 208(2) can be adjusted to a duration that is less than the pulse duration 214 associated with the reference frame time 212 via the light output adjustment 220(1). Controlling the duration of the light output for the light pulse 208(2) can include driving the light emitting element 206 for a shorter amount of time than the amount of time the light emitting element 206 is driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, the control of the light output (or light output adjustment 220(1)) is performed digitally by setting a digitized value to control the light output of the light pulse 208(2).

[0040] During the relatively lower refresh rate toward the right side of the timeline 210 and at the time between the light pulse 208(N-1) and 208(N), the logic of the display system can determine an Nth time difference 218(N) between the illumination of the light pulse 208(N-1) or the light emitting element 206 for the previous frame in the series of frames and the illumination of the upcoming light pulse 208(N) or the light emitting element 206 for the current frame. Based at least in part on the Nth time difference 218(N), which is indicative of the instantaneous refresh rate, the logic can determine a value of a light output parameter that can correspond to or be indicative of the width of the light pulse 208(N) for the current frame. Thus, this value of the light output parameter determined based on the time difference 218(N) can be used to control the light emitting element 206 to emit the light pulse 208(N) for a particular duration. This particular duration of the light pulse 208(N) is shown to be longer than the pulse duration 214 associated with the target luminance level for the reference frame time 212. In other words, because the refresh rate toward the right side of the timeline 210 is lower than the refresh rate corresponding to the reference frame time 212 in the middle of the timeline 210, the height or amplitude of the light pulse 208(N) can be adjusted to a width and duration that is greater than the pulse duration 214 associated with the reference frame time 212 via the light output adjustment 220(N). Controlling the duration of the light output for the light pulse 208(N) can include driving the light emitting element 106 for a longer amount of time than the amount of time the light emitting element 206 is driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, the control of the light output (or light output adjustment 220(N)) is performed digitally by setting a digitized value to control the light output of the light pulse 208(N). Notably, the display driving circuit has sufficient (or extra) bandwidth to provide the ability to increase the pulse width. Figure 2

[0040] During the relatively lower refresh rate toward the right side of the timeline 210 and at the time between the light pulse 208(N-1) and 208(N), the logic of the display system can determine an Nth time difference 218(N) between the illumination of the light pulse 208(N-1) or the light emitting element 206 for the previous frame in the series of frames and the illumination of the upcoming light pulse 208(N) or the light emitting element 206 for the current frame. Based at least in part on the Nth time difference 218(N), which is indicative of the instantaneous refresh rate, the logic can determine a value of a light output parameter that can correspond to or be indicative of the width of the light pulse 208(N) for the current frame. Thus, this value of the light output parameter determined based on the time difference 218(N) can be used to control the light emitting element 206 to emit the light pulse 208(N) for a particular duration. This particular duration of the light pulse 208(N) is shown to be longer than the pulse duration 214 associated with the target luminance level for the reference frame time 212. In other words, because the refresh rate toward the right side of the timeline 210 is lower than the refresh rate corresponding to the reference frame time 212 in the middle of the timeline 210, the height or amplitude of the light pulse 208(N) can be adjusted to a width and duration that is greater than the pulse duration 214 associated with the reference frame time 212 via the light output adjustment 220(N). Controlling the duration of the light output for the light pulse 208(N) can include driving the light emitting element 106 for a longer amount of time than the amount of time the light emitting element 206 is driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, the control of the light output (or light output adjustment 220(N)) is performed digitally by setting a digitized value to control the light output of the light pulse 208(N). Notably, the display driving circuit has sufficient (or extra) bandwidth to provide the ability to increase the pulse width.

[0041] Similarly, reference frame time 212 can be used to determine the corresponding optical output parameters. Figure 2 The value of the duration of the light pulse 108 shown. For example, the logic of the display system can determine the ratio of the first time difference 218(1) to the reference frame time 212, and this ratio can be used to determine the value of the light output parameter for the light output adjustment 220(1) associated with the light pulse 208(2). Consider the following example: where the first time difference 218(1) is approximately 8.3 ms, which corresponds to a refresh rate of 120 Hz, and where the reference frame time 212 is approximately 11.11 ms, which corresponds to a refresh rate of 90 Hz. In this example, the following ratio can be calculated: In this example, the light pulse 208(2) can be adjusted to approximately 75% of the pulse duration 214 associated with the target brightness level for the reference frame time 212. Because pulses 208(1) and 208(2) are relatively close together in time, the shortening of the pulse duration compensates for the increase in brightness caused by the more closely spaced light pulses 208 at a relatively high refresh rate, which maintains the constant brightness of the HMD 200.

[0042] Consider the following example: where the Nth time difference 218(N) is approximately 16.66 ms, corresponding to a refresh rate of 60 Hz, and the reference frame time 212 remains approximately 11.11 ms. In this example, the following ratios can be calculated: In this example, the light pulse 208(N) can be adjusted to approximately 150% of the pulse duration 214 associated with the target brightness level for the reference frame time 212. Because pulses 208(N-1) and 208(N) are time-separated, the extended pulse duration compensates for the brightness reduction caused by the more dispersed light pulses 208 at a relatively low refresh rate, thus maintaining the constant brightness of the HMD200.

[0043] Figure 1 and Figure 2 The corresponding methods shown can be combined in some implementation schemes. For example, the amplitude and duration of the light pulse 108 can be dynamically adjusted by determining a first value of a first light output parameter corresponding to the amplitude of the light pulse 108 / 208 and a second value of a second light output parameter corresponding to the duration of the light pulse 108 / 208, wherein the light-emitting element is illuminated according to both the first value and the second value to adjust the light output.

[0044] In implementations in which the display system uses a scrolling band type of display driving scheme, the light output can be adjusted at least in part by varying the thickness of the scrolling band that traverses the display panel 104 / 204 during the illumination time period for a given frame. That is, the value of the light output parameter can correspond to the number of light emitting elements that are illuminated at the same time (e.g., the number of rows that make up the illuminated scrolling band). While a display system using a global flash type of display driving scheme can adjust the duration of the light pulse 108 / 208, a scrolling band equivalent to adjusting the duration of the light pulse can adjust (e.g., increase or decrease) the number of light emitting elements that are illuminated at the same time during the illumination time period. A "thicker" scrolling band can effectively shorten the duration of the light pulse, while a "thinner" scrolling band can effectively lengthen the duration of the light pulse.

[0045] It should be appreciated that determining the time difference 118 / 218 between a pair of sequential light pulses 108 / 208 corresponding to a pair of sequential frames can include determining the time difference between any two corresponding points of the pair of light pulses 108 / 208 or the pair of frames. For example, the time difference 118 / 218 between the start of each light pulse 108 / 208, the midpoint of each light pulse 108 / 208, the end of each light pulse 108 / 208, etc. can be determined. In some implementations, the display system logic can include frame start markers in the pixel data between each application-rendered frame, and these frame start markers can be used to determine the time difference for determining the value of the light output parameter.

[0046] Figure 3 FIGS. 3 1 (1) and 310(2) illustrate exemplary timelines 310(1) and 310(2) for rendering a series of frames 302 with re-projection at variable frame rates and presenting corresponding images at variable refresh rates, in accordance with implementations disclosed herein. Figure 3 The example depicts three exemplary frames 302(1) (or frame "F"), 302(2) (or frame "F+1"), and 302(3) (or frame "F+2") with respect to the rendering timeline 310(1) to illustrate how the frames 302 can be rendered in sequence. Here, the application 304 renders the frame F first, then the frame F+1, and then the frame F+2 in sequence from left to right on the rendering timeline 310(1). The rendering timeline 310(1) also shows the rendering workload 306 of the compositor 308 of the HMD 100 / 200 (or display system) towards the end of each rendering interval for each frame 302. The application 304 takes a certain amount of time to render each frame 302. This amount of time is denoted as "T" in the example. The rendering workload 306 of the compositor 308 is also shown towards the end of each rendering interval for each frame 302. The compositor 308 takes a certain amount of time to render each frame 302. This amount of time is denoted as "C" in the example. Figure 3The time is represented as application rendering time 312. Because the frame rate can vary, the application rendering time 312(1) for frame 302(1) (or frame “F”), the application rendering time 312(2) for frame 302(2) (or frame “F+1”) and the application rendering time 312(3) for frame 302(2) (or frame “F+2”) can be different amounts of time. Figure 3 An example is shown in which the application rendering time 312 gradually increases from frame F to frame F+2.

[0047] The individual rendering workloads 306 of compositor 308 for a given frame 302 may represent adjustments applied to pixel data output by application 304 before rendering the final image on HMD 100 / 200. Rendering workload 306 may be approximately 1% to 5% of the rendering workload of application 304, meaning that application 304 typically takes longer to render frame 302 compared to when compositor 308 applies adjustments to the pixel data output by application 304 during its rendering workload 306. Such adjustments may include, but are not limited to, adjustments to chroma distortion, panel masking, reprojection, etc., applied to frame 302 output by application 304 before rendering the final image on HMD 100 / 200. Application 304 may represent a video game application or any other type of graphics-based application. Application 304 may execute in a graphics pipeline that outputs pixel data, and compositor 308 is configured to modify that pixel data and output the modified pixel data to a frame buffer (e.g., a stereo frame buffer).

[0048] Before application 304 begins rendering a given frame 302, logic in the display system (e.g., compositor 308) determines a predicted illumination time 314 for the given frame 302, which represents the time during which the light-emitting elements 106 / 206 will illuminate the given frame 302. The actual illumination times 314(1), 314(2), and 314(3) for frames 302(1), 302(2), and 302(3), respectively, are shown on the "scan output + illumination" timeline 310(2). The task of compositor 308 (or other logic) is to predict the illumination time 314 for each frame, which is variable due to the variable frame rate and variable refresh rate. The compositor 308's prediction of the illumination time 314 may be based on historical application rendering times 312. For example, compositor 308 may look at the previous frame (frame F-1, in...)... Figure 3application rendering time 312 for frame F, and can assume that the application rendering time 312(1) for frame F will be the same as the previous frame. The compositor 308 can add the fixed scan-out time 316 to the predicted application rendering time 312 to determine the predicted illumination time 314.

[0049] The predicted illumination time 314 is used, along with head tracking data generated by the head tracking system of the HMD 100 / 200, to determine a predicted pose that the HMD 100 / 200 will be in at the predicted illumination time 314 for a given frame 302. The compositor 308 sends pose data to the application 304 indicating the predicted pose for rendering frame 302, and receives pixel data for frame 302 from the application 304 when the compositor finishes rendering frame 302. Providing the pose data to the application 304 in advance allows the application 304 to output pixel data for rendering an image on the HMD 100 / 200 in a manner that is correct for the predicted head pose of the user 102 / 202 at the future predicted illumination time 314. This means that when light from the display panel 104 / 204 reaches the eyes of the user 102 / 202, the application 304 presents a scene that is appropriate for the predicted head pose of the user at the illumination time 314. In other words, the application 304 is able to render a frame in advance of the predicted illumination time 314, and the compositor 308 is able to apply re-projection adjustments to the pixel data for the frame to account for the fact that the HMD 100 / 200 will not be in the predicted pose at the predicted illumination time 314. This is described in more detail below. Figure 3 In the example of FIG. 3, after the application 304 renders frame F, the compositor 308 can determine the actual application rendering time 312(1) for frame F, and can update its prediction of the illumination time 314, and based on the updated prediction of the illumination time 314, can determine a new predicted pose of the HMD 100 / 200. Based on the initial predicted pose of the HMD 100 / 200 and the new predicted pose of the HMD 100 / 200 (e.g., by comparing the two sets of pose data), the compositor 308 can determine re-projection adjustments, and apply the re-projection adjustments to the pixel data during the rendering workload 306(1) to modify the pixel data for frame F. The modified pixel data for frame F is output to the frame buffer, and scanned out during the scan-out time 316(1) to present an image corresponding to frame F (as modified by the re-projection adjustments) on the display panel 104 of the HMD 100 / 200 at the illumination time 314(1). During the scan-out time 316, a subset of the pixel values (pixel data) for frame F are scanned out to the display panel 104 via a display port (e.g., a high-definition multimedia interface (HDMI)) in sequence. During the illumination time 314, the light emitting elements 106 / 206 of the display panel 104 are illuminated to cause the pixels to emit light. The illumination can be a global flash in which all of the light emitting elements 106 / 206 are illuminated simultaneously, or a rolling illumination in which a subset of the light emitting elements 106 / 206 are illuminated in sequence. In either case, the illumination is considered herein to be a light pulse 108 / 208 for a given frame.

[0050] The graphics logic of the HMD 100 / 200 (or display system) can be asynchronous or synchronous. In an asynchronous system, the compositor 308 runs on a graphics processing unit (GPU) of the HMD 100 / 200 (display system or HMD system) separately from the application 304 (on a separate asynchronous thread). For example, the application 304 can call a function that receives pose data from the compositor 308, and the compositor 308 can provide the requested (predicted from the predicted time of illumination 314(1) for frame F) pose data to the application 304 so that the application 304 can render frame 302(1) (e.g., frame F) from that pose data, which corresponds to the virtual camera pose used to render the scene. Because variable refresh rates are supported, the compositor's workload 306 begins after the application 304 completes rendering frame 302 (the application render time 312 is variable). The compositor 308 can be configured to fetch frame 302 (e.g., left and right images) from the application 304 and distort frame 302 into a back buffer on the display panel 104 / 204 of the HMD 100 / 200. During the compositor's 308 workload 306, re-projection adjustments can be applied.

[0051] By dynamically matching the refresh rate to the frame rate of the application 304, a newly rendered frame 302 is received from the application 304 for each screen refresh, and the images corresponding to a given frame 302 can be presented with only small-scale re-projection adjustments, if any. If the predicted application render time 312 is accurate, the degree of re-projection adjustment will be minimized. Even if the prediction of the application render time 312 is inaccurate, and thus the pose prediction is inaccurate, the amount of re-projection will be small due to the variable refresh rate capability of the display system, which means that the excess visual artifacts relative to moving objects or animated display objects caused by re-projection will be mitigated.

[0052] The processes described herein are illustrated as a collection of blocks in logical flow graphs, which represent a sequence of operations (i.e., logical) that can be implemented in hardware, software, firmware, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement the processes.

[0053] Figure 4A flowchart showing an exemplary process 400 for rendering frames with re-projection in a low-visual-persistence, variable-refresh-rate display system, in accordance with embodiments disclosed herein, is shown. For purposes of discussion, the process 400 is described with reference to the preceding figures.

[0054] At 402, a frame 302 of a series of frames can be rendered for presenting a corresponding image on a display panel 104 / 204 of the HMD 100 / 200. For purposes of discussion, this frame 302 is referred to as a first frame 302(1). If the HMD 100 / 200 includes a pair of display panels, the first frame 302(1) can be rendered for presenting a first image on a left display panel 104 / 204 and a second image on a right display panel 104 / 204, or vice versa. As shown by various sub-frames within 402, there can be various sub-operations performed by logic of the HMD 100 / 200 (or HMD system) to achieve the purpose of rendering the first frame 302(1) at block 402.

[0055] At sub-frame 404, logic of the HMD 100 / 200 (or HMD system), e.g., including logic of the compositor 308, can determine a predicted illumination time 314(1) for the first frame 302(1). Prior to this illumination time prediction, the application 304 can make a call to the compositor 308 requesting a pose of the HMD 100 / 200 so that the application 304 can render the first frame 302(1) appropriately for this HMD pose. The predicted illumination time 314 determined at sub-frame 404 can represent a time at which the light-emitting element 106 / 206 will be illuminated for the first frame 302(1). In other words, the illumination time prediction at sub-frame 404 is a prediction of when light will actually reach the user's eye 102 / 202 when presenting an image corresponding to the first frame 302(1) on the display panel 104 / 204 of the HMD 100 / 200. As shown by sub-frames 406 and 408 within sub-frame 404, there can be various sub-operations performed by logic of the HMD 100 / 200 (or HMD system) in order to determine the predicted illumination time 314(1) at sub-frame 404.

[0056] At sub-frame 406, logic of the HMD 100 / 200 (or HMD system), e.g., including logic of the compositor 308, can determine a first amount of time taken by the application 304 to render a previous frame 302.

[0057] At sub-block 408, logic can determine a predicted render time 312(1) representing an amount of time that application 304 will take to render first frame 302(1), based at least in part on the first amount of time determined at sub-block 406 (as application render times 312 are variable over a series of frames 302, depending on scene complexity and / or current load on processing resources, etc.). In one example, if application 304 took 11 ms to render a previous frame 302, the predicted render time 312(1) determined at sub-block 408 can be the same amount of time: 11 ms. This is based on the concept that it is reasonably safe to assume that it will take as long to render an upcoming frame 302 as it took to render a previous frame 302. The predicted render time 312(1) determined at sub-block 408 can be in the form of a predicted time at which application 304 will complete rendering first frame 302(1), or the predicted render time can be in the form of a predicted amount of time that application 304 will take to render first frame 302(1) (e.g., from start to finish). Thus, sub-blocks 406 and 408 illustrate how a prediction of application render time 312(1) for first frame 302(1) can be based on one or more historical render times 312 of previous frames 302 (i.e., frames 302 prior to first frame 302(1)). However, it should be understood that a prediction of application render time 312(1) can be based on various other factors, such as scene complexity, current processing load, etc.

[0058] In short, while the application render time 312(1) (i.e., the amount of time the application 304 takes to render the first frame 302(1)) is variable, the scan-out time 316(1) for the first frame 302(1) is fixed, and the scan-out time 316(1) plus the illumination time 314(1) can correspond to the fastest frame rate in a range of possible frame rates. Thus, if the frame rate is pushed to the high end for a variable refresh rate display, e.g., 144 Hz, the pixel data is scanned out and an image for each frame 302 is presented on the display approximately every 6.9 ms. In other words, a given frame 302 should take approximately 6.9 ms (for a display system capable of achieving a 144 Hz refresh rate on the high end) to scan out pixel data from the frame buffer and illuminate the pixels on the display. The illumination time 314(1) (technically, a period of time) that occurs after the pixel data is scanned out is very short compared to both the application render time 312(1) and the scan-out time 316(1). This illumination time 314(1) also includes a settling time. For example, for an LCD, the crystals typically need an additional few milliseconds to settle, so the light emitting elements 106 / 206 are illuminated after the crystals have settled. Some or all of these periods of time can be taken into account in order to determine the predicted illumination time 314(1) at the sub-frame 404. At least because the frame rate is variable, the predicted illumination time 302(1) is determined based at least in part on a predicted application render time 312(1) for the first frame 314(1). In some embodiments, such as for a rolling band type of display driving scheme, the illumination time prediction at the sub-frame 404 can be based on a prediction of where the user 102 / 202 will be on the display panel 104 / 204, which can be determined from eye tracking data generated by an eye tracking system of the HMD 100 / 200.

[0059] At the sub-frame 406, in some embodiments, the logic can determine the application render times 312 for a number of previous frames 302, and can determine an average application render time 312 based on these historical render times to predict the application render time 312(1) for the first frame 302(1). For example, the logic can determine a first amount of time that the application 304 took to render a previous frame 302, and a second amount of time that a previous frame 302 took before the previous frame 302 was rendered by the application 304, and can then determine an average render time 312 based on the first and second amounts of time. The number of previous frames 302 to consider for this running average render time calculation is configurable (e.g., the average render time 312 is calculated based on the render times 312 of the ten most recent rendered (previous) frames 302). Employing such an averaging approach can provide the benefit of not reacting to abnormal render times, but waiting for the render times to settle out a bit before adjusting the illumination time prediction.

[0060] At sub-block 410, logic (e.g., compositor 308) can determine a predicted pose that HMD 100 / 200 will be in at predicted illumination time 314(1), determined at sub-block 404. This is an initial pose prediction for HMD 100 / 200 for first frame 302(1), and it can be based at least in part on head tracking data generated by a head tracking system of HMD 100 / 200 (or HMD system).

[0061] At sub-block 412, logic (e.g., compositor 308) can send pose data to application 304 indicating the pose predicted at sub-block 410 for the purpose of rendering first frame 302(1).

[0062] At sub-block 414, logic (e.g., compositor 308) can receive pixel data for first frame 302(1) from application 304. As described herein, the pixel data can include pixel values for individual pixels in the pixel array of display panel 104 / 204.

[0063] At sub-block 416, logic (e.g., compositor 308) can determine a new predicted illumination time 314(1) representing the time at which emissive element 106 / 206 will be illuminated for first frame 302(1) based at least in part on the actual amount of time it took application 304 to render first frame 302(1). The actual rendering time 312 for first frame 302(1) (which can be determined at sub-block 418) can be less than the initially predicted rendering time 312, in which case compositor 308 receives pixel data from application 304 faster than predicted. The actual rendering time 312 for first frame 302(1) can be greater than the initially predicted rendering time 312, in which case compositor 308 receives pixel data from application 304 later than predicted. The predicted application rendering time 312 can be accurate, meaning that the pixel data is received accurately when predicted to be received, but in many cases there can be an increment between the predicted rendering time and the actual rendering time for first frame 302(1), although the increment can be small.

[0064] At sub-block 420, logic (e.g., compositor 308) can determine a new predicted pose that HMD 100 / 200 will be in at new predicted illumination time 314(1) based at least in part on head tracking data generated by a head tracking system of HMD 100 / 200. The increment between the initial pose prediction and the new pose prediction depends on the accuracy of the application rendering time 312 prediction. However, as indicated, the increment can be small given the relatively short playback period.

[0065] At sub-block 422, logic (e.g., compositor 308) can apply a re-projection adjustment to the pixel data for the first frame 302(1) to obtain modified pixel data associated with the first frame 302(1). In Figure 3 In the diagram of FIG. 4, this can occur during the rendering workload 306(1) after the compositor 308 receives the pixel data for the first frame 302(1) from the application 304. The re-projection adjustment can be determined based at least in part on the initial predicted pose (determined at sub-block 410) and the new predicted pose (determined at sub-block 420), such as by comparing the two sets of pose data to determine an offset. In the illustrative example, the compositor 308 can receive the pixel data for the first frame 302(1) one or two milliseconds earlier than predicted, or one or two milliseconds later than predicted. Because the pixel data is to be output to a frame buffer and scanned out to a display as quickly as possible, the updated prediction of the illumination time 314(1) can be slightly different (e.g., different by an amount on the order of one or two milliseconds) than the initial prediction of the illumination time 314(1), and the HMD pose prediction can be updated based on this small increment to determine the re-projection adjustment to be made to the pixel data.

[0066] At sub-block 424, logic (e.g., compositor 308) can output the modified (first) pixel data for the first frame 302(1) to a frame buffer. Again, for an HMD 100 / 200 having a pair of display panels 104 / 204, this pixel data can correspond to a frame representing a pair of images to be displayed on the pair of display panels 104 / 204. As indicated by off-page reference "A" in FIG. 4, the process 400 can continue to Figure 5 the first block of process 500 shown in FIG. 5.

[0067] Figure 5 A flowchart illustrating an exemplary process 500 for dynamically determining values for light output parameters used to illuminate a low-persistence, variable refresh rate display during image rendering in accordance with embodiments disclosed herein is shown. As Figure 4 and Figure 5 As indicated by off-page reference "A" in FIG. 4, the process 500 can continue from the process 400. The process 500 can also be performed independently based on any pixel data output to a frame buffer. For purposes of discussion, the process 500 is described with reference to the preceding figures.

[0068] At 502, logic (e.g., display controller, display driver circuit, etc.) of a display system, such as an HMD 100 / 200 (or HMD system), can cause an image to be rendered on a display based at least in part on pixel data for a frame 302 rendered by an application 304, possibly with re-projection, as described in Figure 4 the process 400 of FIG. 4. For purposes of discussion, and continuingFigure 4 In the example discussed in FIG. 3, the image is referred to as a first image corresponding to the first pixel data of the first frame 302(1). As shown in sub-blocks 504 and 506, rendering the image at block 502 can include one or more sub-operations.

[0069] At sub-block 504, for example, the modified (first) pixel data for the first frame 302(1) can be scanned out on the display panel 104 / 204 of the display system. For example, the pixel values of the pixels of each row can be sequentially scanned out to the display panel 104 / 204 via a display port (e.g., HDMI), starting with the pixels of the first subset (e.g., row) and ending with the pixels of the last subset (e.g., row).

[0070] At sub-block 506, the light-emitting elements 106 / 206 of the display can be illuminated during the rendering of the first image on the display. Illumination can occur by emitting a light pulse 108 / 208. That is, the low- persistence display system can pulse the light-emitting elements 106 / 206 on and off over a series of frames at a refresh rate of the display, which can be variable as described herein. The light pulse during illumination at sub-block 506 can be global flash-style illumination (in which the light-emitting elements 106 / 206 are illuminated simultaneously) or rolling band-style illumination (in which subsets of the light-emitting elements 106 / 206 are illuminated sequentially during the light pulse 108 / 208), as described herein. As shown in sub-blocks 508-514, values of light output parameters for illuminating the light-emitting elements 106 / 206 during the rendering of the first image can be determined such that the luminance of the display remains constant over the series of frames.

[0071] At sub-block 508, logic of the display system (e.g., HMD 100 / 200) can determine a first time difference 118(1) / 218(1) between a light pulse 108(0) (or illumination) of the light emitting elements 106 / 206 for a previous frame 302 in the series of frames and an upcoming light pulse 108(1) (or illumination) of the light emitting elements 106 / 206 for the first frame 302(1). This first time difference 118(1) / 218(1) can be referred to herein as a current frame time, and this first time difference can be derived from a current refresh rate of the display. In an illustrative example, consider that the first time difference 118(1) / 218(1) (or current frame time) is 8.3 ms, which indicates an instantaneous refresh rate of 120 Hz. It should be understood that determining the first time difference 118(1) / 218(1) between a pair of sequential light pulses 108 / 208 corresponding to a pair of sequential frames 302 can include determining a time difference between any two corresponding points between the pair of light pulses 108 / 208 or the pair of frames. For example, a time difference 118 / 218 between a start of each light pulse 108 / 208, a midpoint of each light pulse 108 / 208, an end of each light pulse 108 / 208, etc. can be determined. In some embodiments, the display system logic can include frame start markers in the pixel data between each application-rendered frame, and these frame start markers can be used to determine the first time difference 118(1) / 218(1) at sub-block 508.

[0072] At sub-block 510, logic of the display system (e.g., HMD 100 / 200) can determine a first value of a light output parameter based at least in part on the first time difference 118(1) / 218(1). As described herein, a value of the light output parameter can correspond to an amplitude of the light pulse 108(1) for the first frame 302(1). A value of another light output parameter can correspond to a duration of the light pulse 108(1) for the first frame 302(1). In a rolling band type of display driving scheme, a value of the light output parameter can correspond to a number of light emitting elements 106 / 206 that are illuminated simultaneously during the light pulse 108(1) (e.g., a thickness of the illuminated rolling band). In some embodiments, multiple ones of these light output parameters can be determined and applied in combination to control the light output.

[0073] At sub-block 512, as part of determining the first value of the light output parameter, the logic can determine a ratio of the first time difference 118(1) / 218(1) to the reference frame time 112 / 212. As described herein, the reference frame time 112 / 212 can be a time corresponding to an intermediate frame rate between a minimum frame rate and a maximum frame rate in a frame rate range to which the application 304 is targeted. For example, if the application 304 is targeted to a frame rate range between 45 FPS and 144 FPS, the reference frame time used to determine the aforementioned ratio can correspond to a frame time at a frame rate of 90 FPS, which is a frame time of approximately 11.11 ms. In this example, the reference frame time 112 / 212 would be 11.11 ms. However, this is merely an example, and the reference frame time 112 / 212 is configured based in part on the specifications of the display system in which the techniques described herein are implemented. In the illustrative example, if the first time difference 118(1) / 218(1) (or the current frame time) is 8.3 ms, and the reference frame time 112 / 212 is 11.11 ms, then the ratio determined at sub-block 512 would be approximately 0.75. This can be considered as the light output factor being 75% of the nominal light output at the reference frame rate 112 / 212, which means that the light output will be reduced in some manner (whether it is a reduction in the amplitude or duration of the light pulse 108(1), etc.). In any case, the first value of the light output parameter can be determined based at least in part on this ratio.

[0074] At sub-block 514, the light emitting element 106 / 206 can be controlled to emit light pulses 108(1) (or illuminate) in accordance with the first value of the light output parameter during the presentation of the first image on the display. The process 500 can iterate over a series of frames by dynamically determining a value of the light output parameter for each frame in the series of frames 302. As a result of varying the light output parameter in this manner over the series of frames 302, the display brightness remains substantially constant over the series of frames, thereby eliminating or at least mitigating flicker of the display.

[0075] Figure 6 An example component of an HMD 600 (or an HMD system including the HMD 600) is shown, such as an embeddable VR headset, in accordance with the embodiments disclosed herein. The HMD 600 can be the same as or similar to the HMD 100 / 200 referenced in the previous figures, and thus can be implemented in the HMD 100 / 200 Figure 6components of the illustrated HMD 600. The HMD 600 can be implemented as a standalone device to be worn by the user 102 (e.g., worn on the head of the user 102). In some embodiments, the HMD 600 can be head-mountable, such as by allowing the user 102 to secure the HMD 600 on his / her head using a securing mechanism (e.g., an adjustable band) sized to fit around the head of the user 102. In some embodiments, the HMD 600 includes a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye display. As such, the terms “wearable device,” “wearable electronic device,” “VR headset,” “AR headset,” and “head-mounted display (HMD)” can be used interchangeably herein to refer to the device 600. Figure 6 However, it should be understood that these types of devices are merely examples of the HMD 600, and it should be understood that the HMD 600 can be implemented in various other form factors. It should also be understood that some or all of the illustrated components can be implemented on the HMD 600. Thus, in some embodiments, a subset of the illustrated components can be implemented on a computing device that is part of the HMD system but separate from the HMD 600 itself, such as a PC, game console, or any other suitable computing device. It should also be understood that a non-HMD system can include some similar components to at least implement the dynamic light output adjustment techniques and processes described herein. Figure 6 Thus, in some embodiments, a subset of the illustrated components can be implemented on a computing device that is part of the HMD system but separate from the HMD 600 itself, such as a PC, game console, or any other suitable computing device. It should also be understood that a non-HMD system can include some similar components to at least implement the dynamic light output adjustment techniques and processes described herein. Figure 6 Thus, in some embodiments, a subset of the illustrated components can be implemented on a computing device that is part of the HMD system but separate from the HMD 600 itself, such as a PC, game console, or any other suitable computing device. It should also be understood that a non-HMD system can include some similar components to at least implement the dynamic light output adjustment techniques and processes described herein.

[0076] In the illustrated implementation, the HMD 600 includes one or more processors 602 and a memory 604 (e.g., a computer-readable medium 604). In some implementations, the processor 602 can include a central processing unit (CPU), a graphics processing unit (GPU) 603, both a CPU and a GPU 603, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processors 602 can have its own local memory that can also store program modules, program data, and / or one or more operating systems.

[0077] Memory 604 can include volatile memory and non-volatile memory implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by computing device. Memory 604 can be implemented as a computer readable storage medium (“CRSM”) which can be any available physical medium that is accessible by processor 602 to execute the instructions stored on memory 602. In one basic implementation, the CRSM can include both RAM and Flash memory. In other implementations, the CRSM can include, but is not limited to, read-only memory (“ROM”), electrically programmable read-only memory (“EPROM”), or electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium which can be used to store the desired information and which can be accessed by processor 602.

[0078] Generally, HMD 600 (or HMD system) can include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. Computer readable medium 604 is shown including various modules, such as instructions, data stores, etc., which can be configured to execute on processor 602 to implement the techniques, functions, and / or operations described herein. Some example functional modules are shown as being stored in computer readable medium 604 and executable on processor 602, but the same functionality can alternatively be implemented in hardware, firmware, or a system on a chip (SOC), and / or other logic.

[0079] For the benefit of other modules, the operating system module 606 can be configured to manage hardware located within and coupled to the HMD 600. Additionally, in some instances, the HMD 600 can include one or more application programs 304 stored in the memory 604 or otherwise accessible by the HMD 600. In this implementation, the application programs 304 include a game application 608. However, the HMD 600 can include any number or type of application programs and is not limited to the specific examples shown here. The game application 608 can be configured to initiate a game start of a video-based interactive game (e.g., a VR game) that can be played by the user 102 and output frames 302 to be rendered on the display panel of the HMD 600. In combination with other logic of the HMD 600, the compositor 308 can be configured to perform the techniques described herein to render the frames 302 with re-projection, such as by implementing the process 400. In combination with other logic of the HMD 600, the light output adjustment component 610 can be configured to perform the techniques described herein to dynamically adjust the light output of the light emitting elements 106 / 206 based on the time difference between sequential light pulses 108 / 208 corresponding to sequential frames 302.

[0080] Generally, the HMD 500 has input devices 612 and output devices 614. The input devices 612 can include control buttons. In some implementations, one or more microphones can be used as input devices 612 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) can be used as input devices 612 to receive pose input, such as user 102 hand and / or head movements. In some embodiments, additional input devices 612 can be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, hand controller, etc. In other embodiments, the HMD 600 can omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 600 can be implemented with a relatively simple form of input devices 612, a network interface (wired or wireless based), power, and processing / storage capabilities. For example, a limited set of one or more input components (e.g., a dedicated button for initiating configuration, power on / off, etc.) can be employed so that the HMD 600 can be used thereafter. In one implementation, the input devices 612 can include control mechanisms such as a base volume control button for increasing / decreasing volume, as well as power and reset buttons.

[0081] The output device 614 can include a display 616, which can include one or more display panels 104 / 204 (e.g., a stereoscopic display panel 104 / 204 pair) that include light emitting elements 106 / 206 as described herein. The output device 614 can also include, but is not limited to, light emitting elements (e.g., LEDs), a vibrator for generating haptics, a speaker (e.g., earphones), etc. There can also be a simple light emitting element (e.g., LED) to indicate status, such as when powered on, for example.

[0082] The HMD 600 can also include a wireless unit 618 coupled to an antenna 620 to facilitate wireless connectivity with a network. The wireless unit 618 can implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the HMD 600 can also include physical ports to facilitate wired connectivity with a network, connected peripheral devices (including a PC, game console, etc.), or plug-in network devices (which can be part of the HMD system) that communicate with other wireless networks.

[0083] The HMD 600 can also include an optical subsystem 622 that uses one or more optical elements to direct light from the electronic display 616 to the user’s eye. The optical subsystem 622 can include different types of different optical elements and combinations thereof, including but not limited to, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more of the optical elements in the optical subsystem 622 can have one or more coatings, such as an anti-reflective coating. Magnification of the image light by the optical subsystem 622 allows the electronic display 616 to be physically smaller, lighter in weight, and consume less power compared to larger displays. Additionally, magnification of the image light can increase the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., 120 to 150 degrees diagonal) and in some cases all of the user’s FOV. AR applications can have a narrower FOV (e.g., about 40 degree FOV). The optical subsystem 622 can be designed to correct for one or more optical errors, such as but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, the content provided to the electronic display 616 for display is pre-distorted, and the optical subsystem 622 corrects for this distortion as it receives image light from the electronic display 616 that is generated based on the content.

[0084] The HMD 600 can also include one or more sensors 624, such as sensors to generate motion, position, and orientation data. These sensors 624 can be or include gyroscopes, accelerometers, magnetometers, cameras, color sensors, or other motion, position, and orientation sensors. The sensors 624 can also include subparts of sensors, such as a series of active or passive markers that can be externally observed by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset can include a plurality of markers on its exterior, such as reflectors or lights (e.g., infrared or visible lights), that can provide one or more reference points for software to interpret in order to generate motion, position, and orientation data when observed by an external camera or illuminated by light (e.g., infrared or visible light). The HMD 600 can include light sensors that are sensitive to light (e.g., infrared or visible light) projected or scattered into the environment of the HMD 600 by a base station.

[0085] In one example, the sensors 624 can include an inertial measurement unit (IMU) 626. The IMU 626 can be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors or some combination thereof adapted to detect motion, correct for errors associated with the IMU 626. Based on the measurement signals, such motion-based sensors, such as the IMU 626, can generate calibration data that indicates an estimated position of the HMD 600 relative to an initial position of the HMD 600. For example, multiple accelerometers can measure translational motion (forward / back, up / down, left / right), and multiple gyroscopes can measure rotational motion (e.g., pitch, yaw, and roll). The IMU 626 may, for example, sample the measurement signals rapidly and compute an estimated position of the HMD 600 from the sampled data. For example, the IMU 626 can integrate the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 600. A reference point is a point that can be used to describe a position of the HMD 600. While a reference point can generally be defined as a point in space, in various implementations, the reference point is defined as a point within the HMD 600 (e.g., the center of the IMU 626). Alternatively, the IMU 626 provides the sampled measurement signals to an external console (or other computing device) that determines the calibration data.

[0086] The sensors 624 can operate at a relatively high frequency in order to provide sensor data at a high rate. For example, the sensor data can be generated at a rate of 1000 Hz (or one sensor reading taken every 1 millisecond). As such, one thousand readings are taken per second. When the sensors generate this much data at this rate (or at a greater rate), the data set for predicting motion is quite large, even for a relatively short period of time of about tens of milliseconds.

[0087] As mentioned, in some embodiments, the sensors 624 can include light sensors that are sensitive to light emitted by base stations in the environment of the HMD 600 for the purpose of tracking the position and / or orientation, pose, etc. of the HMD 600 in 3D space. The calculation of position and / or orientation can be based on the timing characteristics of the light pulses and the presence or absence of light detected by the sensors 624.

[0088] HMD 600 can also include an eye tracking system 628 that generates eye tracking data. The eye tracking system 628 can include, without limitation, a camera or other optical sensor internal to the HMD 600 to capture image data (or information) of the user's eyes, and the eye tracking system 628 can use the captured data / information to determine a motion vector, an interpupillary distance, an interocular distance, a three-dimensional (3D) position of each eye relative to the HMD 600, including the magnitude of the twist and rotation (i.e., roll, pitch, and yaw) of each eye, and a gaze direction. In one example, infrared light is emitted within the HMD 600 and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system 628 and analyzed to extract eye rotation from changes in the infrared light reflected from each eye. The eye tracking system 628 can use a number of methods for tracking the eyes of the user 102. Thus, the eye tracking system 628 can track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw), and at least a subset of the quantities tracked from both eyes of the user 102 can be combined to estimate a gaze point (i.e., a 3D positioning of where the user is looking or a location in a virtual scene), which can be mapped to a location on the display panel 104 / 204 for predicting where the user 102 will be looking in terms of individual subsets (e.g., rows) or groups of contiguous subsets (e.g., groups of contiguous rows) of pixels of the display panel 104 / 204. For example, the eye tracking system 628 can integrate information from past measurements, measurements that identify the position of the head of the user 102, and 3D information that describes a scene presented by the electronic display 616. Thus, information for the position and orientation of the eyes of the user 102 is used to determine a gaze point in a virtual scene presented by the HMD 600 that the user 102 is looking at and map that gaze point to a location on the display panel 104 / 204 of the HMD 600. The motion vector can be used to predict a trajectory of eye movement, etc.

[0089] HMD 600 can also include a head tracking system 630. The head tracking system 630 can utilize one or more of the sensors 624 to track head motion (including head rotation) of the user 102, as described above. For example, the head tracking system 630 can track up to six degrees of freedom of the HMD 600 (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at each frame 302 in a series of frames 302 so that the application 304 can determine how to render the scene in the next frame 302 according to the head position and orientation, and thus the re-projection adjustment calculations can be made by the compositor 308. In some embodiments, the head tracking system 630 and / or the compositor 308 using the head tracking system 630 are configured to predict future poses (positions and / or orientations) of the HMD 600 based on current and / or past data and / or based on known / implicit scan-out delays of pixels of various subsets of display systems implementing the rolling band display driving techniques, as described herein. This is because the application 304 is required to render the frames 302 before the user 102 actually sees the light on the display 616 (and thus the image). Thus, the next frame 302 can be rendered based on this future prediction of the head position and / or orientation made at an earlier point in time, which can be in the range of approximately 12 to 30 milliseconds (ms) before the frame 302 is rendered, depending on the instantaneous frame rate and / or refresh rate. The rotational data provided by the head tracking system 630 can be used to determine both a direction of rotation of the HMD 600 and an amount of rotation of the HMD 600 in any suitable unit of measurement. For example, the direction of rotation can be simplified and output according to positive or negative horizontal directions and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation can be in terms of degrees, radians, etc. An angular velocity can be calculated to determine a rate of rotation of the HMD 600.

[0090] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A display system comprising: a head-mounted display, the display having an array of light emitting elements, the head-mounted display configured to support a variable refresh rate; a head tracking system; one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the display system to: determine a first predicted illumination time, the first predicted illumination time representing a time at which the light emitting elements are to be illuminated for a first frame in a series of frames, wherein determining the first predicted illumination time comprises: determining a first amount of time taken by an application to render a previous frame in the series of frames; determining a second amount of time taken by the application to render a frame prior to the previous frame; determining an average rendering time based at least in part on the first amount of time and the second amount of time; and determining a first predicted rendering time representing an amount of time that the application is to take to render the first frame based at least in part on the average rendering time; wherein the first predicted illumination time is determined based at least in part on the first predicted rendering time; determine a first predicted pose at which the head-mounted display is to be at the first predicted illumination time based at least in part on first head tracking data generated by the head tracking system; send first pose data indicative of the first predicted pose to the application for rendering the first frame; receive pixel data for the first frame from the application; determine a second predicted illumination time representing the time at which the light emitting elements are to be illuminated for the first frame based at least in part on an amount of time taken by the application to render the first frame; determine a second predicted pose at which the head-mounted display is to be at the second predicted illumination time based at least in part on the first head tracking data generated by the head tracking system; apply a re-projection adjustment to the received pixel data for the first frame based at least in part on the first predicted pose and the second predicted pose to obtain first pixel data associated with the first frame; output the first pixel data associated with the first frame to a frame buffer; determine a first time difference between illumination of the light emitting elements for the previous frame and upcoming illumination of the light emitting elements for the first frame; determine a first value of a light output parameter based at least in part on the first time difference; render a first image on the head-mounted display based at least in part on the first pixel data; and illuminate the light emitting elements according to the first value of the light output parameter during rendering of the first image on the head-mounted display.

2. The display system of claim 1, wherein the computer-executable instructions, when executed by the one or more processors, further cause the display system to: output second pixel data associated with a second frame in the series of frames to the frame buffer; ​ determining a second time difference between illumination of the light emitting element for the first frame and upcoming illumination of the light emitting element for the second frame; determining a second value of the light output parameter based at least in part on the second time difference; rendering a second image on the head-mounted display based at least in part on the second pixel data; and illuminating the light emitting element according to the second value of the light output parameter during the rendering of the second image on the head-mounted display, wherein: the second time difference is greater than the first time difference; and the second value of the light output parameter causes the light emitting element to emit light at a higher brightness or for a longer duration than the first value of the light output parameter, or wherein: the second time difference is less than the first time difference; and the second value of the light output parameter causes the light emitting element to emit light at a lower brightness or for a shorter duration than the first value of the light output parameter.

3. The display system of claim 1, wherein the first value of the light output parameter corresponds to at least one of: an amplitude of a light pulse used to illuminate the light emitting element; a duration of the light pulse used to illuminate the light emitting element; or a number of the light emitting elements that are simultaneously illuminated at the same time as individual subsets of the light emitting elements are sequentially illuminated during the rendering of the first image on the head-mounted display.

4. The display system of claim 1, wherein determining the first value of the light output parameter based at least in part on the first time difference comprises: determining a ratio of the first time difference to a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

5. The display system of claim 1, wherein the application program is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, wherein the computer-executable instructions, when executed by the one or more processors, further cause the display system to determine a ratio of the first time difference to a reference frame time, and wherein the reference frame time corresponds to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

6. The display system of claim 1, wherein the display system is at least one of a virtual reality (VR) display system or an augmented reality (AR) display system.

7. A method implemented by a display system, wherein the display system comprises: a head-mounted display having an array of light emitting elements, the head-mounted display configured to support a variable refresh rate; and a head tracking system; the method comprising: determining a first predicted illumination time, the first predicted illumination time representing a time at which a light pulse for a first frame of a series of frames will occur, wherein determining the first predicted illumination time comprises: determining a first amount of time taken by an application program to render a previous frame of the series of frames; determining a second amount of time taken by the application program to render a frame preceding the previous frame; ​ ​ determining an average render time based at least in part on the first amount of time and the second amount of time; and determining a first predicted render time representing an amount of time that the application will take to render the first frame based at least in part on the average render time; wherein the first predicted illumination time is determined based at least in part on the first predicted render time; determining a first predicted pose that the head-mounted display will be in at the first predicted illumination time based at least in part on first head tracking data generated by the head tracking system; sending first pose data indicative of the first predicted pose to the application for rendering the first frame; receiving pixel data for the first frame from the application; determining a second predicted illumination time representing the time at which the light pulse for the first frame will occur based at least in part on an amount of time that the application took to render the first frame; determining a second predicted pose that the head-mounted display will be in at the second predicted illumination time based at least in part on the first head tracking data generated by the head tracking system; applying a re-projection adjustment to the received pixel data for the first frame based at least in part on the first predicted pose and the second predicted pose to obtain first pixel data associated with the first frame; outputting the first pixel data to a frame buffer; determining a first time difference between a light pulse of the light emitting elements for the previous frame and an upcoming light pulse of the light emitting elements for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the head-mounted display based at least in part on the first pixel data; and controlling the light emitting elements to emit light pulses according to the first value of the light output parameter during the presentation of the first image on the head-mounted display.

8. The method of claim 7, wherein the first value of the light output parameter corresponds to at least one of: an amplitude of the light pulse for the first frame; a duration of the light pulse for the first frame; or a number of the light emitting elements that are simultaneously illuminated during the light pulse for the first frame.

9. The method of claim 7, wherein determining the first value of the light output parameter based at least in part on the first time difference comprises: determining a ratio of the first time difference to a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

10. The method of claim 7, wherein the controlling the light emitting elements to emit the light pulses comprises at least one of: causing the light emitting elements to emit light simultaneously; or causing individual subsets of the light emitting elements to emit the light sequentially. ​ 11. The method of claim 7, wherein the application program is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, wherein the method further comprises determining a ratio of the first time difference to a reference frame time, and wherein the reference frame time corresponds to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

12. The method of claim 7, wherein the head-mounted display is a virtual reality (VR) headset.

13. The method of claim 7, wherein the application program is a video game application program.

Images