Display system with dynamic light output adjustment for maintaining constant brightness

Abstract

To dynamically adjust light output of a display on-the-fly.SOLUTION: When implemented on a low-persistence display that supports a variable refresh rate, this adjustment maintains a constant brightness over a series of frames to eliminate flickering of the display. When pixel data of a given frame is output to a frame buffer for presenting an image on the display, a time difference between an illumination of the display's light emitting elements for a preceding frame and an upcoming illumination of the light emitting elements for the given frame may be determined, and this time difference is used to determine a value of a light output parameter. During the presentation of the image, the light emitting elements can be illuminated in accordance with the value of the light output parameter. This determination is iterated over a series of frames to dynamically adjust the display's light output.SELECTED DRAWING: Figure 1

Description

Cross - reference to related applications

[0001] This is a PCT application claiming priority to U.S. Patent Application No. 16 / 399,804, titled "DISPLAY SYSTEM WITH DYNAMIC LIGHT OUTPUT ADJUSTMENT FOR MAINTAINING CONSTANT BRIGHTNESS", filed on April 30, 2019, which is hereby incorporated by reference in its entirety.

Background Art

[0002] Conventional displays for virtual reality (VR) systems (such as those embedded in VR headsets) operate at a fixed refresh rate. The "refresh rate" of a display is the number of times per second that the display can refresh an image or redraw the screen (e.g., a 90-hertz (Hz) display refreshes the image 90 times per second). Advancements in graphics have led to the development of displays that support a variable refresh rate, which means that the refresh rate of the display changes dynamically as frames are rendered. This allows the refresh rate to be maintained in sync with the various frame rates from graphics rendering applications such as video games. For example, both G-SYNC™, a technology available from NVIDIA® Corporation, and FreeSync™, a technology available from Advanced Micro Devices®, Inc., provide logic to match the refresh rate of the display to the frame rate of a video game. These technologies eliminate screen tearing (e.g., when an image contains some pixels from a previous frame and some pixels from the current frame), and these technologies allow a running video game to target a range of frame rates instead of a single frame rate, which may allow the video game to benefit from the fastest possible frame rate considering the speed of the graphics processing unit (GPU).

[0003] Variable refresh rate techniques are suitable for use in high persistence displays where the backlight is on most of the time over a series of frames, but these techniques are not suitable for use in VR systems because most VR systems use low persistence displays that pulse the backlight on and off at the display's refresh rate. When variable refresh rate is used in a low persistence display, the viewing user will notice flicker of the display as the brightness changes with the change in refresh rate. This is because at higher refresh rates, the light pulses occur closer together in time, creating a brightening effect, while at lower refresh rates, the light pulses occur further apart in time, creating a dimming effect. This appears as continuous brightening and dimming (i.e., flicker) of the display to a user wearing a head-mounted display (HMD). In practice, a scene with low complexity (e.g., a scene with few moving objects and simple textures) is likely to have a higher refresh rate for the low complexity scene, so it will appear very bright to the user, while a scene with high complexity (e.g., a scene with many moving objects and complex textures) is likely to have a lower refresh rate for the high complexity scene, so it will appear very dim to the user. Therefore, variable refresh rate display technology was not suitable for use in VR systems using low persistence displays.

[0004] This specification provides technical solutions for improving and enhancing these systems and other systems.

Brief Description of the Drawings

[0005] The detailed description will be explained with reference to the accompanying drawings. In the figures, the leftmost digit of the reference number identifies the figure in which the reference number first appears. The use of the same reference number in different figures indicates similar or identical components or features.

[0006]

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

DETAILED DESCRIPTION OF THE INVENTION

[0012] In this specification, in particular, techniques and systems for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames will be described. The display system may include a display having an array of light-emitting elements (or light sources). As described above, the display system may be a low-persistence display system, meaning that the light-emitting elements emit light for only a very small portion of the frame time. For example, for a given frame at a refresh rate of 90 Hz, the light-emitting elements can illuminate (or emit light pulses) for a duration of approximately 1 millisecond (ms) out of the total frame time, which is approximately 11.11 ms. In an illustrative example, the display of the display system disclosed herein may be a head-mounted display (HMD) that can be worn by a user. In this example, the display system may be a virtual reality (VR) system or an augmented reality (AR) system. The display itself, such as an HMD, may include one or more display panels that present images based on frames output by a graphics rendering application such as a video game. These images are viewed by the user through the optics included in the HMD, causing the user to perceive the images as if the user were immersed in a VR or AR environment.

[0013] The displays described in this specification may support variable refresh rates. Thus, graphics rendering applications such as video games may render frames at variable frame rates, and the refresh rate of the display may be dynamically adjusted to match (or remain in sync with) the frame rate of the application. In this way, the application can target a range of frame rates, and the refresh rate of the display can be tied to the frame rate of the application such that a single frame is rendered and the corresponding image is presented with each screen refresh. Since the refresh rate can change dynamically, if the light output of the light-emitting elements remains constant over each light pulse, the brightness of the display can vary over a series of frames, which can be perceived by the user as display flicker.

[0014] Accordingly, this specification describes techniques and systems for maintaining a constant brightness of a low persistence display that supports variable refresh rates. The logic of the disclosed display system can be configured to output first pixel data to a frame buffer for the purpose of presenting a corresponding first image on the display, where the first pixel data is associated with a first frame of a series of frames. The logic can determine a first time difference between the illumination (or light pulse) of the light emitting elements of the display for a previous frame of the series of frames and the next illumination (or light pulse) of the light emitting elements for the first frame corresponding to the first image presented. Based on this first time difference (which indicates the instantaneous refresh rate of the display), the logic of the display system can determine a first value of a light output parameter. During the presentation of the first image on the display, the light emitting elements can be controlled to illuminate according to the first value of the light output parameter. This process can be repeated over a series of frames by dynamically determining the value of the light output parameter for each frame, and as a result of varying the light output parameter over the series of frames in this way, the brightness of the display remains substantially constant over the series of frames, thereby eliminating or at least reducing flicker.

[0015] Generally, when the time difference between a pair of consecutive light pulses corresponding to a pair of consecutive frames is relatively short (meaning a relatively high refresh rate), the logic of the display system determines the value of the light output parameter that causes a decrease in the magnitude or duration of the light output from the light-emitting element. On the other hand, when the time difference between a pair of consecutive light pulses is relatively long (meaning a relatively low refresh rate), the logic of the display system determines the value of the light output parameter that causes an increase in the magnitude or duration of the light output from the light-emitting element. In this way, a constant brightness is achieved by increasing the magnitude and / or duration of the light output at a relatively low refresh rate and decreasing the magnitude and / or duration of the light output at a relatively high refresh rate.

[0016] As described above, adjusting the light output of the light-emitting element of the display can be achieved in a plurality of ways. For example, the pulse of light for a given frame can be made higher than the height of the previous frame or reference pulse (by controlling the light-emitting element to emit light at a higher luminous intensity, for example). In the reverse direction, the pulse of light for a given frame can be made shorter than the height of the previous frame or reference pulse (by illuminating the light-emitting element at a lower luminous intensity, for example). Further, or alternatively, to make the pulse of light higher or shorter, the pulse of light for a given frame can be made wider than the width of the previous frame or reference pulse (by illuminating the light-emitting element over a longer duration, for example). In the reverse direction, the pulse of light for a given frame can be made narrower than the width of the previous frame or reference pulse (by illuminating the light-emitting element over a shorter duration, for example). To maintain a constant brightness, the area under the pulse (e.g., the area under the square waveform representing the pulse) may remain constant regardless of the dynamic adjustment to the height and / or width of the pulse (or the representative waveform of the pulse).

[0017] In some embodiments, the reference frame time is associated with a target level of brightness at which the display remains constant. This reference frame time can be used to determine the value of the light output parameter. For example, when determining the difference in time between consecutive light pulses corresponding to a pair of consecutive frames, the logic can determine the ratio of the time difference to the reference frame time, and use this ratio to determine the value of the light output parameter for adjusting the light output of the light emitting element for the current frame. The reference frame time may be the time corresponding to an intermediate frame rate between the minimum frame rate and the maximum frame rate within the range of frame rates targeted by the application. For example, if the application targets a frame rate range of 45 frames per second (FPS) to 144 FPS, the reference frame time used to determine the aforementioned ratio may correspond to the frame time at a frame rate of 90 FPS, which is approximately 11.11 ms. This is merely an example, and the reference frame time can be configured based in part on the specifications of the display system in which the techniques described herein are implemented.

[0018] Techniques and systems for applying "reprojection" adjustment to low-persistence variable refresh rate display systems such as display systems including HMDs are also described herein. Using the described reprojection techniques, it is possible to compensate for a slight inaccuracy in the prediction of the original pose of the HMD based on the predicted illumination time for a given frame before the given frame is rendered. Reprojection can be used to modify pixel data received from an application for a given frame, for example, by converting the pixel data (e.g., through rotation and reprojection calculations) in a way that takes into account the updated prediction of the pose of the HMD after the actual rendering time of the frame has been allowed for the display system. Unwanted visual artifacts can be reduced by using reprojection to present the correct pixels at the time when light from the display reaches the user's eyes.

[0019] By dynamically adjusting the light output based on the time difference between consecutive pulses of light for a pair of consecutive frames (the time difference indicating the instantaneous frame rate), the display flicker (i.e., brightness fluctuations) over a series of frames can be reduced, if not eliminated. This can be done independently of or in combination with reprojection. Combining dynamic light output adjustment with the reprojection techniques described herein provides a more robust display system that maintains a constant brightness of the display while reducing visual artifacts. In other words, with the techniques and systems described herein, a display system can present the correct scene at the location the user is currently viewing while maintaining a constant brightness of the display.

[0020] The techniques and processes disclosed herein may be implemented by a system such as an HMD system. Also disclosed herein is a non-transitory computer-readable medium storing computer-executable instructions for implementing the techniques and processes disclosed herein. The techniques and systems disclosed herein are considered in the context of, by way of example, a video game application, specifically a VR game application, but it should be understood that the techniques and systems described herein may provide benefits to other applications, including but not limited to non-VR applications (e.g., AR applications), and / or non-game applications such as industrial machine applications, defense applications, robotic applications. Further, the techniques and processes described herein may be implemented in non-HMD systems such as other display systems that are not considered “near-to-eye” display systems or that do not involve wearable devices. Additionally, specific advantages are described for low persistence display systems that support variable refresh rates, but it should be understood that the techniques and processes described herein can be implemented with other types of display systems, and the techniques and processes described herein are not necessarily limited to low persistence display systems or variable refresh rate display systems.

[0021] Figure 1 is a diagram illustrating an exemplary technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames, according to an embodiment disclosed herein. Figure 1 depicts a head-mounted display (HMD) 100 worn by a user 102. HMD 100 is an example of a display system that may implement the techniques and processes described herein. Thus, HMD 100 may sometimes be referred to herein simply as a "display" or a "display system." The HMD 100 of the example of Figure 1 may include a single display panel 104 or multiple display panels 104, such as a left display panel and a right display panel of a stereo pair of display panels. One or more display panels 104 of HMD 100 may be used to present a series of image frames (hereinafter referred to as "frames") visible to a user 102 wearing HMD 100. It should be understood that HMD 100 may include any number of display panels 104 (e.g., three or more display panels, a pair of display panels, or a single display panel). Thus, the term "display panel," when used herein in the singular, may refer to a single display panel 104 of a display system having any number of display panels (e.g., "display panel" may refer to either display panel 104 of a pair of display panels). For example, in a 2-panel HMD 100, a stereo frame buffer may render, for example, 2160 × 1200 pixels (e.g., 1080 × 1200 pixels per display panel) on both display panels of HMD 100. For example, in a 2-panel HMD 100, a stereo frame buffer may render, for example, 2160 × 1200 pixels (e.g., 1080 × 1200 pixels per display panel) on both display panels of HMD 100.

[0022] The HMD100 can utilize any suitable type of display technology, such as a light-emitting display, which emits light using light-emitting elements (e.g., light-emitting diodes (LEDs)) during the presentation of frames on the display panel 104. As an example, the display panel 104 of the HMD100 can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0023] The display panel 104 of the HMD100 can support a variable refresh rate. Thus, the HMD100 may be operable over any suitable range of refresh rates, such as in the range of 45 Hertz (Hz) to 144 Hz, 45 Hz to 120 Hz, 90 Hz to 144 Hz, or any other suitable range. The "refresh rate" of a display is the number of times per second that the display can refresh an image or redraw the screen. The number of frames displayed per second can be equivalent to the instantaneous refresh rate of the display. In other words, when a series of frames is being processed (e.g., rendered) and the image is being presented on the display, the logic of the HMD100 can aim to present an image for a single frame out of the series of frames for each screen refresh.

[0024] The HMD100 can implement any suitable low-persistence driving method, including but not limited to a "global flashing" type of display driving method or a "rolling band" type of display driving method. Using global flashing technology, all light-emitting elements 106 (e.g., an M×N array of light-emitting elements 106 that may be referred to herein as "light sources" 106) can be illuminated synchronously to present a single image on the display for a given frame by illuminating all pixels of the display. Using the rolling band technique, individual subsets of the light-emitting elements 106, and thus individual subsets of the pixels, can be sequentially illuminated independently within a rolling band of illumination during the illumination period. This rolling band technique may be enabled by the light-emitting elements 106 being individually addressable. In some embodiments, both the pixel array and the light-emitting elements 106 on the display panel 104 are arranged in rows and columns, but not necessarily one pixel per one light-emitting element 106 correspondence. In this configuration, in the case of a rolling band type of display driving method, individual rows and / or individual columns of the light-emitting elements 106 can be sequentially addressed, and / or individual groups of consecutive rows and / or individual groups of consecutive columns of the light-emitting elements 106 can be sequentially addressed. As a result of addressing the light-emitting elements 106 in this "rolling" manner, individual subsets of the pixels corresponding to these individually addressable subsets of the light-emitting elements can be independently "illuminated".

[0025] As used herein, "illuminating a pixel" means illuminating the light-emitting element 106 corresponding to that pixel. For example, an LCD illuminates the light-emitting element 106 of a backlight to illuminate the corresponding pixel of the display. Further, as used herein, a "subset of pixels" can include individual pixels or a plurality of pixels (e.g., a group of pixels). Similarly, a "subset of light-emitting elements 106" can include individual light-emitting elements 106 or a plurality of light-emitting elements 106 (e.g., a group of light-emitting elements 106). In some embodiments, a subset of pixels includes a row of pixels, a column of pixels, a group of consecutive rows of pixels, or a group of consecutive columns of pixels. Similarly, a subset of light-emitting elements 106 includes a row of light-emitting elements 106, a column of light-emitting elements 106, a group of consecutive rows of light-emitting elements 106, or a group of consecutive columns of light-emitting elements 106. Thus, in aspects of the techniques and systems described herein, a subset of pixels can be sequentially scanned and illuminated, such as by sequentially illuminating each row of pixels starting from a first row of pixels (e.g., the top row of pixels) and ending at a last row of pixels (e.g., the bottom row of pixels) with a corresponding subset of light-emitting elements. However, any suitable illumination pattern (e.g., a lighting pattern like a snake, column-by-column illumination, sequentially illuminating a plurality of rows / columns of pixels or light-emitting elements 106 at once, etc.) can be used using the techniques and systems described herein.

[0026] To drive the display panel 104, the HMD 100 can include, among other things, a display controller such as a microcontroller or a similar processor, a display driver circuit, and similar electronics for driving the display panel 104. The display driver circuit can be connected to an array of light-emitting elements 106 of the display panel 104 via a conductive path such as a metal trace on a flexible printed circuit. In one embodiment, the display controller can be communicatively coupled to the display driver circuit and configured to provide signals, information, and / or data to the display driver circuit. The signals, information, and / or data received by the display driver circuit can include one or more light output parameters that cause the light-emitting elements 106 to be illuminated in a particular manner by the display driver circuit. That is, the display controller can determine which light-emitting elements 106 are illuminated, when the elements 106 are illuminated, the magnitude of the light output (e.g., the height of a pulse), and / or the duration of the light output (e.g., the width of a pulse), etc., and the display controller can communicate appropriate signals, information, and / or data (e.g., one or more light output parameters) to the display driver circuit to achieve that purpose.

[0027] The pixel data for a given frame can be output to the HMD 100, or to a frame buffer for presenting the frame as an image on the display panel 104 of the HMD 100. The pixel data for each frame can include, in some embodiments, a two-dimensional array of per-pixel values (e.g., color values). In some embodiments, the pixel data further includes additional data or metadata such as depth values. In some embodiments, the pixel data can include data for each pixel represented by a set of a single color and alpha value (e.g., one color value for the red channel, one color value for the green channel, one color value for the blue channel, and one or more values for one or more alpha channels). This pixel data can be output to a frame buffer (e.g., a stereo frame buffer) to present a desired visual effect in an image on the display panel 104 of the HMD 100.

[0028] The HMD 100 can represent a VR headset for use in a VR system, such as for use in a VR game system. However, the HMD 100 can additionally or alternatively be implemented as an AR headset for use in an AR application. In AR, the user 102 sees virtual objects overlaid on the real-world environment, whereas in VR, the user 102 does not see the real-world environment and is fully immersed in a virtual environment as perceived through the display panel 104 and optical system (e.g., lenses) of the HMD 100. The embodiments described herein mainly relate to the VR-based HMD 100, but it should be understood that the HMD 100 is not limited to implementation in VR applications, nor is the display system limited to implementation in the HMD 100.

[0029] Generally, a graphics-based application (e.g., a video game) executed on a computing device such as the HMD 100 itself or a computing device associated with and coupled to the HMD 100 as part of a display / HMD system (e.g., a personal computer (PC), a game console, etc.) can be configured to output a series of frames. The series of frames are ultimately presented as images on the display panel 104 of the HMD 100. During the presentation of a single image corresponding to a single frame, a low-persistence display system is configured to pulse the light-emitting elements 106 of the display panel 104, such that, as a result of emitting light towards the eyes of the user 102, the user 102 can view the image presented on the display during the pulsing of the light 108. The timeline 110 shown in FIG. 1 illuminates a series of light pulses 108(1), 108(2), … 108(N) (collectively 108) that can occur at a variable refresh rate of the display system. Each of these light pulses 108 represents an individual emission of light from the light-emitting element 106 for a given frame. The series of light pulses 108 is offset from, but synchronized with, the rendering of a series of frames by an application due to the logic that matches the refresh rate of the HMD 100 to the frame rate of the application rendering the frames.

[0030] As shown by the timeline 110 of FIG. 1, the HMD 100 transitions over time from a relatively high refresh rate (towards the left side of the timeline 110) to a relatively low refresh rate (towards the right side of the timeline 110). In the illustrative example, the refresh rate towards the left side of the timeline 110 may be on the order of 144 Hz, and the refresh rate towards the right side of the timeline 110 may be on the order of 60 Hz. In this regard, the timeline 110 of FIG. 1 is not to scale but is simply illustrative of the variable refresh rate in general.

[0031] FIG. 1 shows a reference frame time 112 (which may also be referred to as the “nominal” frame time) that can be associated with a target level of brightness with the display remaining constant. Considering an example where the reference frame time 112 is approximately 11.11 ms, corresponding to a refresh rate of 90 Hz. In this example, if the refresh rate remains constant at 90 Hz, the light-emitting element 106 is illuminated consistently over a series of frames according to the values of the light output parameters corresponding to a light pulse 108 having a pulse duration 114 and a pulse magnitude 116, as shown in the center of the timeline 110 in FIG. 1. The light pulse 108 having a pulse duration 114 and a pulse magnitude 116 at the refresh rate corresponding to the reference frame time 112 achieves the target level of brightness with the display remaining constant. Thus, as the refresh rate changes, light output adjustment is dynamically performed to keep the brightness constant at this target level.

[0032] During a relatively high refresh rate towards the left side of timeline 110, and at the time between light pulse 108(1) and light pulse 108(2), the logic of the display system can determine a first time difference 118(1) between light pulse 108(1) (i.e., the illumination of light emitting element 106 for a previous frame in a series of frames) and the next light pulse 108(2) (i.e., the next illumination of light emitting element 106 for the current frame). Based at least in part on the first time difference 118(1) (indicating the instantaneous refresh rate), the logic can determine a value of the light output parameter. In the embodiment of FIG. 1, the light output parameter corresponds to or requires the height of light pulse 108(2) for the current frame. Thus, using this value of the light output parameter determined based on the time difference 118(1), the light emitting element 106 can be controlled to emit a light pulse 108(2) of a specific magnitude. This specific magnitude of light pulse 108(2) is shown in FIG. 1 as being smaller than the magnitude 116 of the pulse associated with the target level of brightness at the reference frame time 112. In other words, since the refresh rate towards the left of timeline 110 is higher than the refresh rate corresponding to the reference frame time 112 in the center of timeline 110, the height or magnitude of light pulse 108(2) can be adjusted to a magnitude smaller than the magnitude 116 of the pulse associated with the reference frame time 112 via the light output adjustment 120(1). That is, the dashed outline represents the "reference pulse" in terms of the pulse magnitude 116 and the pulse duration 114, and the solid outline represents the actual light pulse 108(2), and the reference pulse is shown to illustrate the magnitude of light pulse 108(2) relative to the magnitude 116 of the reference pulse. Controlling the luminous intensity of the light output for light pulse 108(2) can include driving the light emitting element 106 with a current and / or voltage lower than the current and / or voltage that can be used to drive the light emitting element 106 at the refresh rate corresponding to the reference frame time 112.In some embodiments, the control of the light output (or light output adjustment 120(1)) is implemented in a digital manner by setting a digitized value to control the light output of the light pulse 108(2).

[0033] During a relatively low refresh rate towards the right side of timeline 110, and at the time between light pulse 108(N - 1) and light pulse 108(N), the logic of the display system can determine the Nth time difference 118(N) between light pulse 108(N - 1) (or the illumination of the light emitting element 106 for the previous frame in a series of frames) and the next light pulse 108(N) (or the next illumination of the light emitting element 106 for the current frame). Based at least in part on the Nth time difference 118(N) (indicating the instantaneous refresh rate), the logic can determine a value of a light output parameter that can correspond to or require the height of light pulse 108(N) for the current frame. Thus, using this value of the light output parameter determined based on the time difference 118(N), the light emitting element 106 can be controlled to emit a light pulse 108(N) of a specific magnitude. This specific magnitude of the light pulse 108(N) is shown in FIG. 1 as being greater than the magnitude 116 of the pulse associated with the target level of brightness of the reference frame time 112. In other words, since the refresh rate towards the right of the timeline 110 is lower than the refresh rate corresponding to the reference frame time 112 in the center of the timeline 110, the height or magnitude of the light pulse 108(N) can be adjusted to a magnitude greater than the magnitude 116 of the pulse associated with the reference frame time 112 via the light output adjustment 120(N). Controlling the luminous intensity of the light output for the light pulse 108(N) can include driving the light emitting element 106 with a current and / or voltage that is higher than the current and / or voltage that can be used to drive the light emitting element 106 at the refresh rate corresponding to the reference frame time 112. In some embodiments, the control of the light output (or light output adjustment 120(N)) is implemented in a digital manner by setting a digitized value to control the light output of the light pulse 108(N). In particular, the display drive circuit has a sufficient (or extra) bandwidth to provide the ability to increase the height of the pulse.

[0034] In some embodiments, the reference frame time 112 is used to determine the value of the light output parameter corresponding to the magnitude of the light pulse 108 shown in FIG. 1. For example, the logic of the display system may determine the ratio of the first time difference 118(1) to the reference frame time 112, and use this ratio to determine the value of the light output parameter for the light output adjustment 120(1) associated with the light pulse 108(2). Consider an example where the first time difference 118(1) is approximately 8.3 ms corresponding to a refresh rate of 120 Hz, and the reference frame time 112 is approximately 11.11 ms corresponding to a refresh rate of 90 Hz. In this example, the following ratio can be calculated. [Number] In this example, the light pulse 108(2) can be adjusted to approximately 75% of the magnitude of the pulse magnitude 116 associated with the target level of brightness relative to the reference frame time 112. Since the pulses 108(1) and 108(2) are relatively close to each other in time, the downward adjustment of the pulse magnitude compensates for the increase in brightness caused by the more densely packed light pulses 108 at a relatively high refresh rate to maintain a constant brightness for the HMD 100.

[0035] Consider an example where the Nth time difference 118(N) is approximately 16.66 ms corresponding to a refresh rate of 60 Hz, and the reference frame time 112 is still 11.11 ms. In this example, the following ratio can be calculated. [Number] In this embodiment, the light pulse 108(N) can be adjusted to a magnitude that is approximately 150% of the magnitude 116 of the pulse associated with the target level of brightness with respect to the reference frame time 112. Since the pulses 108(N - 1) and 108(N) are relatively far apart in time, the upward adjustment of the pulse magnitude compensates for the decrease in brightness caused by the more dispersed light pulses 108 at a relatively low refresh rate that maintains a constant brightness for the HMD 100.

[0036] FIG. 2 is a diagram illustrating another exemplary technique for dynamically adjusting the light output of a low-persistence variable refresh rate display to maintain a constant brightness over a series of frames, according to an embodiment disclosed herein. Similar to the embodiment of FIG. 1, as shown by the timeline 210 of FIG. 2, the HMD 200 transitions over time from a relatively high refresh rate (towards the left side of the timeline 210) to a relatively low refresh rate (towards the right side of the timeline 210). In the illustrated embodiment, the refresh rate towards the left side of the timeline 210 may be about 144 Hz, and the refresh rate towards the right side of the timeline 210 may be about 60 Hz. In this regard, the timeline 210 of FIG. 2 is also not to scale, but simply illustrates the variable refresh rate generally.

[0037] FIG. 2 also shows here a reference frame time 212, to which the display can be associated with a target level of brightness that remains constant. A light pulse 208 having a pulse duration 214 and a pulse magnitude 216 at the refresh rate corresponding to the reference frame time 212 achieves the target level of brightness at which the display remains constant. Thus, as the refresh rate changes, light output adjustment is dynamically performed to keep the brightness constant at this target level.

[0038] During a relatively high refresh rate towards the left side of the timeline 210, and at the time between light pulse 208(1) and light pulse 208(2), the logic of the display system can determine a first time difference 218(1) between light pulse 208(1) (i.e., the illumination of the light emitting element 206 for a previous frame in a series of frames) and the next light pulse 208(2) (i.e., the next illumination of the light emitting element 206 for the current frame). Based at least in part on the first time difference 218(1) (indicating the instantaneous refresh rate), the logic can determine the value of the light output parameter. In the embodiment of FIG. 2, the light output parameter corresponds to or requires the width of light pulse 208(2) for the current frame. Thus, using this value of the light output parameter determined based on the time difference 218(1), the light emitting element 206 can be controlled to emit light pulse 208(2) over a specific duration. This specific duration of light pulse 208(2) is shown in FIG. 2 as being shorter than the pulse magnitude 214 associated with the target level of brightness of the reference frame time 212. In other words, since the refresh rate towards the left of the timeline 210 is higher than the refresh rate corresponding to the reference frame time 212 in the center of the timeline 210, the width or duration of light pulse 208(2) can be adjusted via the light output adjustment 220(1) to be less than the duration 214 of the pulse associated with the reference frame time 212. Controlling the duration of the light output for light pulse 208(2) can include driving the light emitting element 206 over an amount of time that is shorter than the amount of time the light emitting element 206 can be driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, the control of the light output (or light output adjustment 220(1)) is implemented in a digital manner by setting a digitized value to control the light output of light pulse 208(2).

[0039] During a relatively low refresh rate towards the right side of timeline 210, and at the time between light pulse 208(N - 1) and light pulse 208(N), the logic of the display system can determine the Nth time difference 218(N) between light pulse 208(N - 1) (or the illumination of the light emitting element 206 for the previous frame in a series of frames) and the next light pulse 208(N) (or the next illumination of the light emitting element 206 for the current frame). Based at least in part on the Nth time difference 218(N) (indicating the instantaneous refresh rate), the logic can determine the value of the light output parameter that can correspond to or require the width of light pulse 208(N) for the current frame. Thus, using this value of the light output parameter determined based on the time difference 218(N), the light emitting element 206 can be controlled to emit light pulse 208(N) over a specific duration. This specific duration of light pulse 208(N) is shown in FIG. 2 as being longer than the pulse duration 214 associated with the target level of brightness of the reference frame time 212. In other words, since the refresh rate towards the right of timeline 210 is lower than the refresh rate corresponding to the reference frame time 212 in the center of timeline 210, the height or magnitude of light pulse 208(N) can be adjusted to a width or duration greater than the pulse duration 214 associated with the reference frame time 212 via the light output adjustment 220(N). Controlling the duration of the light output for light pulse 208(N) can include driving the light emitting element 106 for a time period longer than the amount of time the light emitting element 206 can be driven at the refresh rate corresponding to the reference frame time 212. In some embodiments, the control of the light output (or light output adjustment 220(N)) is implemented in a digital manner by setting a digitized value to control the light output of light pulse 208(N). In particular, the display drive circuit has sufficient (or extra) bandwidth to provide the ability to increase the width of the pulse.

[0040] Here too, using the reference frame time 212, the value of the light output parameter corresponding to the duration of the light pulse 108 shown in FIG. 2 can be determined. For example, the logic of the display system can determine the ratio of the first time difference 218(1) to the reference frame time 212, and use this ratio to determine the value of the light output parameter for the light output adjustment 220(1) associated with the light pulse 208(2). Consider an example where the first time difference 218(1) is approximately 8.3 ms corresponding to a refresh rate of 120 Hz, and the reference frame time 212 is approximately 11.11 ms corresponding to a refresh rate of 90 Hz. In this example, the following ratio can be calculated.

Number

[0041] Consider an example where the Nth time difference 218(N) is approximately 16.66 ms corresponding to a refresh rate of 60 Hz, and the reference frame time 212 is still 11.11 ms. In this example, the following ratio can be calculated.

Number

[0042] In some embodiments, the respective approaches shown in FIGS. 1 and 2 can be combined. For example, by determining a first value of a first light output parameter corresponding to the magnitude of the light pulse 108 / 208 and a second value of a second light output parameter corresponding to the duration of the light pulse 108 / 208, the magnitude and duration of the light pulse 108 can be dynamically adjusted, and the light emitting elements can be illuminated according to both the first value and the second value to adjust the light output.

[0043] In embodiments where the display system uses a rolling band type of display driving method, the light output can be at least partially adjusted by changing the thickness of the rolling band that traverses the display panel 104 / 204 during the illumination period of a given frame. That is, the value of the light output parameter can correspond to several light emitting elements that are illuminated simultaneously (e.g., several rows that make up the rolling band of illumination). A display system using a global flashing type of display driving method can adjust the duration of the light pulse 108 / 208, while a rolling band that is equivalent to adjusting the duration of the light pulse can adjust (e.g., increase or decrease) the number of light emitting elements that are illuminated simultaneously during the illumination period. A "thicker" rolling band can effectively shorten the duration of the light pulse, while a "thinner" rolling band can effectively lengthen the duration of the light pulse.

[0044] It should be understood that determining the time difference 118 / 218 between a pair of consecutive light pulses 108 / 208 corresponding to a pair of consecutive frames may include determining the time difference between any two corresponding points between a pair of light pulses 108 / 208 or between a pair of frames. For example, the time difference 118 / 218 can be determined between the start point of each light pulse 108 / 208, the midpoint of each light pulse 108 / 208, and the end point of each light pulse 108 / 208. The logic of the display system may, in some embodiments, include frame start markers in the pixel data between frames rendered by each application, and these frame start markers can be used to determine the time difference for the purpose of determining the value of the light output parameter.

[0045] FIG. 3 is a diagram illustrating exemplary timelines 310(1) and 310(2) for rendering a series of frames 302 by re - projection at a variable frame rate and presenting corresponding images at a variable refresh rate, according to an embodiment disclosed herein. The example of FIG. 3 illustrates three exemplary frames 302(1) (or frame "F"), 302(2) (or frame "F + 1"), and 302(3) (or frame "F + 2") with respect to the rendering timeline 310(1), and explains how the frames 302 can be rendered in series. Here, the application 304 renders the frames F, then the frame F + 1, and then the frame F + 2 sequentially from left to right on the rendering timeline 310(1). The rendering timeline 310(1) also shows the rendering workload 306 of the compositor 308 of the HMD100 / 200 (or display system) towards the end of each rendering interval of each frame 302. The application 304 spends an amount of time to render each frame 302. This amount of time is shown as the application rendering time 312 in FIG. 3. Since the frame rate can vary, the application rendering time 312(1) of frame 302(1) (or frame "F"), the application rendering time 3 12(2) of frame 302(2) (or frame "F + 1"), and the application rendering time 312(3) of frame 302(3) (or frame "F + 2") may be different amounts of time. FIG. 3 shows an example where the application rendering time 312 gradually increases from frame F to frame F + 2.

[0046] For a given frame 302, the individual rendering workload 306 of compositor 308 may represent adjustments applied to the pixel data output by application 304 before the final image is rendered on HMD 100 / 200. The rendering workload 306 may be about 1% to 5% of the rendering workload of application 304, which typically means that it takes much longer for application 304 to render frame 302 than for compositor 308 to apply adjustments to the pixel data output by application 304 during compositor 308's rendering workload 306. Such adjustments may include, but are not limited to, adjustments for chromatic aberration, panel masking, reprojection, etc., applied to frame 302 output by application 304 before the final image is rendered on HMD 100 / 200. Application 304 may represent a video game application, or any other type of graphics-based application. Application 304 may be executed in a graphics pipeline that outputs pixel data, and compositor 308 is configured to modify that pixel data and output the modified pixel data to a frame buffer (e.g., a stereo frame buffer).

[0047] Before application 304 begins rendering a given frame 302, the logic of the display system (e.g., compositor 308) may determine a predicted illumination time 314 for the given frame 302, which represents the time that the light emitting elements 106 / 206 illuminate for the given frame 302. The actual illumination times 314(1), 314(2), and 314(3) for frames 302(1), 302(2), and 302(3) are shown on the "scan out + illumination" timeline 310(2), respectively. The compositor 308 (or other logic) has the task of predicting this illumination time 314 for each frame, which is variable due to variable frame rates and variable refresh rates. The prediction of the illumination time 314 by the compositor 308 may be based on past application rendering times 312. For example, the compositor 308 can examine the application rendering time 312 of a previous frame (frame F-1 not shown in FIG. 3) and assume that the application rendering time 312(1) of frame F is the same as the previous frame. The compositor 308 can determine the predicted illumination time 314 by adding the fixed scan out time 316 to the predicted application rendering time 312.

[0048] This predicted illumination time 314 is used together with head-tracking data generated by the head-tracking system of the HMD 100 / 200 to determine the predicted pose that the HMD 100 / 200 will assume at the predicted illumination time 314 for a given frame 302. The compositor 308 transmits pose data indicating this predicted pose to the application 304 for rendering the frame 302, and receives pixel data for the frame 302 from the application 304 when the rendering of the frame 302 is complete. By providing the pose data to the application 304 in advance, the application 304 can output pixel data for rendering an image on the HMD 100 / 200 in a correct manner with respect to the predicted pose of the user's 102 / 202 head at a future predicted illumination time 314. This means that the application 304 renders a scene appropriate for the predicted pose of the user's head at the illumination time 314 when the light from the display panel 104 / 204 reaches the user's 102 / 202 eyes. In the example of FIG. 3, after the frame F is rendered by the application 304, the compositor 308 may determine the actual application rendering time 312(1) of the frame F, update the prediction of the compositor 308 with respect to the illumination time 314, and based on this updated prediction with respect to the illumination time 314, determine a new predicted pose of the HMD 100 / 200. Based on the original predicted pose of the HMD 100 / 200 and the new predicted pose of the HMD 100 / 200 (e.g., by comparing two sets of pose data), the compositor 308 determines reprojection adjustments and applies these reprojection adjustments to the pixel data during the rendering workload 306(1) to modify the pixel data of the frame F. The modified pixel data of the frame F is output to the frame buffer and scanned out during the scan-out time 316(1) to present an image corresponding to the frame F (modified by the reprojection adjustments) on the display panel 104 of the HMD 100 / 200 at the illumination time 314(1).During the scan-out time 316, a subset of the pixel values (pixel data) of the frame F is sequentially scan-out to the display panel 104 via a display port (e.g., a high-definition multimedia interface (HDMI (registered trademark))). During the illumination time 314, the light-emitting elements 106 / 206 of the display panel 104 are illuminated to illuminate the pixels. The illumination may be global flashing in which all the light-emitting elements 106 / 206 are performed simultaneously, or rolling illumination in which a subset of the light-emitting elements 106 / 206 is performed sequentially. In either case, the illumination is considered herein as a pulse of the light 108 / 208 for a given frame.

[0049] The graphics logic of the HMD100 / 200 (or display system) may be asynchronous or synchronous. In an asynchronous system, the compositor 308 is executed separately (on a separate asynchronous thread) from the application 304 on the graphics processing unit (GPU) of the HMD100 / 200 (display system or HMD system). For example, the application 304 may call a function to receive pose data from the compositor 308, and the compositor 308 provides the application 304 with the requested pose data (predicted for the predicted illumination time 314(1) of the frame F) according to the pose data corresponding to the virtual camera pose used for rendering the scene, so that the application 304 can render the frame 302(1) (e.g., frame F). Since a variable refresh rate is supported, the workload 306 of the compositor starts after the application 304 finishes rendering the frame 302 (the application rendering time 312 is variable). The compositor 308 receives the frame 302 (e.g., left and right images) from the application 304 and may be configured to distort the frame 302 in the back buffer on the display panel 104 / 204 of the HMD100 / 200. During this workload 306 of the compositor 308, reprojection adjustment may be applied.

[0050] By dynamically matching the refresh rate to the frame rate of application 304, newly rendered frames 302 are received from application 304 for each screen refresh, and if an image corresponding to a given frame 302 exists, it can be presented with only minor re-projection adjustments. If the predicted application rendering time 312 is accurate, the degree of re-projection adjustment is minimized. Even with an inaccurate prediction of the application rendering time 312, and thus an inaccurate pose prediction, the variable refresh rate feature of the display system results in a smaller amount of re-projection, which means that undesirable visual artifacts caused by re-projection for moving or animated objects are reduced.

[0051] The processes described herein are presented as a collection of blocks in a logical flow graph and represent a series of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In a software context, the blocks represent computer-executable instructions, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement the process.

[0052] FIG. 4 illustrates a flow diagram of an exemplary process 400 for rendering frames by re-projection in a low-persistence variable refresh rate display system according to an embodiment disclosed herein. For purposes of discussion, process 400 will be described with reference to the previous figure.

[0053] In 402, a certain frame 302 among a series of frames can be rendered to present a corresponding image on the display panel 104 / 204 of the HMD 100 / 200. For the purpose of discussion, this frame 302 is referred to as the first frame 302(1). When the HMD 100 / 200 includes a pair of display panels, the first frame 302(1) can be rendered to present a first image on the left display panel 104 / 204 and a second image on the right display panel 104 / 204, or vice versa. As indicated by various sub-blocks within 402, there can be various sub-operations performed by the logic of the HMD 100 / 200 (or the HMD system) for the purpose of rendering the first frame 302(1) in block 402.

[0054] In sub-block 404, the logic of the HMD 100 / 200 (or the HMD system), for example, the logic including the compositor 308, can determine a predicted illumination time 314(1) for the first frame 302(1). Before this prediction of the illumination time, the application 304 can make a call to the compositor 308 that requests the pose of the HMD 100 / 200, whereby the application 304 can appropriately render the first frame 302(1) with respect to the pose of that HMD. The predicted illumination time 314 determined in sub-block 404 can represent the time during which the light-emitting elements 106 / 206 illuminate for the first frame 302(1). In other words, the prediction of the illumination time in sub-block 404 is a prediction of when the light actually reaches the eyes of the user 102 / 202 when the image corresponding to the first frame 302(1) is presented on the display panel 104 / 204 of the HMD 100 / 200. As indicated by sub-blocks 406 and 408 within sub-block 404, there can be various sub-operations performed by the logic of the HMD 100 / 200 (or the HMD system) for the purpose of determining the predicted illumination time 314(1) in sub-block 404.

[0055] In sub - block 406, the logic of the HMD 100 / 200 (or HMD system), e.g., the logic including compositor 308, may determine a first amount of time that application 304 spent rendering the previous frame 302.

[0056] In sub - block 408, the logic may determine a predicted rendering time 312(1) representing the amount of time that application 304 will spend rendering the first frame 302(1) based at least in part on the first amount of time determined in sub - block 406 (since the application rendering time 312 varies across a series of frames 302 depending on, e.g., the complexity of the scene and / or the current load on the processing resources). In one example, if application 304 spent 11 ms rendering the previous frame 302, the predicted rendering time 312(1) determined in sub - block 408 may be the same amount of time, 11 ms. This is based on the idea that it is reasonably safe to assume that the next frame 302 will take about as long to render as the previous frame 302 took. The predicted rendering time 312(1) determined in sub - block 408 may be in the form of the predicted time at which application 304 will finish rendering the first frame 302(1), or in the form of the predicted amount of time that application 304 will take (e.g., from start to finish) to render the first frame 302(1). Thus, sub - blocks 406 and 408 explain how the prediction of the application rendering time 312(1) for the first frame 302(1) is based on one or more past rendering times 312 of the previous frame 302 (i.e., the frame 302 that came before the first frame 302(1)). However, it should be understood that the prediction of the application rendering time 312(1) may be based on various other factors such as the complexity of the scene, the current processing load, etc.

[0057] Put simply, the application rendering time 312(1) (i.e., the amount of time for the application 304 to render the first frame 302(1)) is variable, but the scan-out time 316(1) for the first frame 302(1) is fixed, and the sum of the scan-out time 316(1) and the illumination time 314(1) can correspond to the fastest frame rate within the range of possible frame rates. Thus, when the frame rate is pushed to 144 Hz at the high end, for example, for a variable refresh rate display, the pixel data is scan-out, and an image is presented on the display for each frame 302 approximately every 6.9 ms. In other words, it should take approximately 6.9 ms to scan out the pixel data from the frame buffer and illuminate the pixels on the display for a given frame 302 (in the case of a display system capable of achieving a 144 Hz refresh rate at the high end). The illumination time 314(1) (technically a period) that occurs after the pixel data has been scan-out is very short compared to both the application rendering time 312(1) and the scan-out time 316(1). The illumination time 314(1) includes the settling time. For example, in the case of an LCD, the crystals often take several extra milliseconds to settle, so the light-emitting elements 106 / 206 are illuminated after the crystals have settled. Some or all of these periods can be considered in determining the predicted illumination time 314(1) in the sub-block 404. At a minimum, since the frame rate is variable, the predicted illumination time 314(1) is determined based at least in part on the predicted application rendering time 312(1) for the first frame 302(1). In some embodiments, in a rolling band type of display driving method, for example, the prediction of the illumination time in the sub-block 404 can be based on a prediction of where the user 102 / 202 is looking on the display panel 104 / 204, which can be determined from the eye-tracking data generated by the eye-tracking system of the HMD 100 / 200.

[0058] In some embodiments, in sub-block 406, the logic may determine the application rendering time 312 for a plurality of previous frames 302 and may determine an average application rendering time 312 based on these past rendering times to predict the application rendering time 312(1) for the first frame 302(1). For example, the logic may determine a first amount of time that the application 304 spent rendering a previous frame 302 and a second amount of time that the application 304 spent rendering a frame 302 that preceded the previous frame 302, and then, based on the first amount of time and the second amount of time, the average rendering time 312 may be determined. The number of previous frames 302 to consider for this running average rendering time calculation is configurable (e.g., calculating the average rendering time 312 based on the rendering times 312 of 10 most recently rendered (previous) frames 302). Taking such an average may provide the advantage of not reacting to abnormal rendering times and instead waiting for the rendering time to settle a bit before adjusting the prediction of the illumination time.

[0059] In sub-block 410, the logic (e.g., compositor 308) may determine a predicted pose that the HMD 100 / 200 will take at the predicted illumination time 314(1) determined in sub-block 404. This is the original prediction of the pose of the HMD 100 / 200 for the first frame 302(1) and may be based at least in part on head tracking data generated by a head tracking system of the HMD 100 / 200 (or HMD system).

[0060] In sub-block 412, the logic (e.g., compositor 308) may send pose data indicating the pose predicted in sub-block 410 to the application 304 for the purpose of rendering the first frame 302(1).

[0061] In sub-block 414, the logic (e.g., compositor 308) may receive pixel data from application 304 regarding the first frame 302(1). The pixel data may include pixel values regarding individual pixels within the array of pixels of display panel 104 / 204, as described herein.

[0062] In sub-block 416, the logic (e.g., compositor 308) may determine a new predicted illumination time 314(1) representing the time for which light-emitting elements 106 / 206 will illuminate for the first frame 302(1), based at least in part on the amount of time application 304 spent rendering the first frame 302(1). The actual rendering time 312 of the first frame 302(1) (which may be determined in sub-block 418) may be less than the original predicted rendering time 312, in which case compositor 308 receives pixel data from application 304 faster than predicted. The actual rendering time 312 of the first frame 302(1) may be greater than the original predicted rendering time 312, in which case compositor 308 receives pixel data from application 304 later than predicted. The predicted application rendering time 312 may have been accurate, meaning that the pixel data was received exactly when it was predicted to be received, but often there may be a small delta between the predicted rendering time and the actual rendering time for the first frame 302(1).

[0063] In sub-block 420, logic (e.g., compositor 308) can determine a new predicted pose that the HMD 100 / 200 will take at the new predicted illumination time 314(1) based at least in part on the head-tracking data generated by the head-tracking system of the HMD 100 / 200. The delta between the prediction of the original pose and the prediction of the new pose depends on the accuracy of the prediction of the application rendering time 312. However, as shown, this delta is likely to be small considering the relatively short period during play.

[0064] In sub-block 422, logic (e.g., compositor 308) can apply reprojection adjustment to the pixel data for the first frame 302(1) to obtain modified (first) pixel data associated with the first frame 302(1). In the figure of FIG. 3, this can occur after receiving the pixel data of the first frame 302(1) from the application 304 during the rendering workload 306(1) of the compositor 308. The reprojection adjustment can be determined based at least in part on the original predicted pose (determined in sub-block 410) and the new predicted pose (determined in sub-block 420), such as by comparing two sets of pose data to determine an offset. In an illustrative example, the compositor 308 can receive the pixel data for the first frame 302(1) one or two milliseconds faster or one or two milliseconds slower than predicted. Since the pixel data is output to the frame buffer and scanned out to the display as fast as possible, the updated prediction of the illumination time 314(1) may be slightly different from the original predicted illumination time 314(1) (e.g., different by about one or two milliseconds), and the pose prediction of the HMD can be updated based on this small delta to determine the reprojection adjustment to be performed on the pixel data.

[0065] In sub-block 424, the logic (e.g., compositor 308) may output modified (first) pixel data for the first frame 302(1) to the frame buffer. Here also, in the case of the HMDs 100 / 200 having a pair of display panels 104 / 204, this pixel data may correspond to a frame representing a pair of images to be displayed on the pair of display panels 104 / 204. As indicated by the off-page reference "A", process 400 may follow the first block of process 500 shown in FIG. 5.

[0066] FIG. 5 illustrates a flowchart of an exemplary process 500 for dynamically determining values of light output parameters for illuminating a low-persistence variable refresh rate display during image presentation, according to an embodiment disclosed herein. As indicated by the off-page references "A" in FIGS. 4 and 5, process 500 may follow from process 400. Process 500 may also be performed independently based on any pixel data output to the frame buffer. For purposes of discussion, process 500 will be described with reference to the previous figure.

[0067] At 502, the logic of a display system, such as the HMD 100 / 200 (or HMD system) (e.g., a display controller, a display driver circuit, etc.), may cause an image to be presented on the display, at least in part, based on pixel data for the rendered frame 302 by the application 304, optionally by reprojection, as described in process 400 of FIG. 4. For purposes of discussion and following the example discussed in FIG. 4, the image is referred to as a first image corresponding to the first pixel data of the first frame 302(1). As indicated by sub-blocks 504 and 506, the presentation of the image at block 502 may include one or more sub-operations.

[0068] In sub-block 504, for example, the modified (first) pixel data for the first frame 302(1) can be scanned out on the display panel 104 / 204 of the display system. For example, the pixel values for the individual rows of pixels can be sequentially scanned out to the display panel 104 / 204 via a display port (e.g., HDMI), starting from the first subset (e.g., row) of pixels and ending with the last subset (e.g., row) of pixels.

[0069] In sub-block 506, the light-emitting elements 106 / 206 of the display can be illuminated during the presentation of the first image on the display. The illumination can be generated by emitting a light pulse 108 / 208. That is, the low persistence display system can pulse the on and off of the light-emitting elements 106 / 206 over a series of frames at the refresh rate of the display, which can be variable as described herein. As described herein, the light pulse during illumination in sub-block 506 can be a global flashing of illumination where the light-emitting elements 106 / 206 are illuminated simultaneously, or a rolling band of illumination where individual subsets of the light-emitting elements 106 / 206 are sequentially illuminated during the light pulse 108 / 208. As indicated by sub-blocks 508 - 514, the value of the light output parameter for illuminating the light-emitting elements 106 / 206 during the presentation of the first image can be determined over a series of frames such that the brightness of the display remains constant.

[0070] In sub - block 508, the logic of the display system (e.g., HMD 100 / 200) can determine a first time difference 118(1) / 218(1) between a light pulse 108(0) (or illumination) of the light - emitting elements 106 / 206 for a previous frame 302 and the next light pulse 108(1) (or illumination) of the light - emitting elements 106 / 206 for the first frame 302(1) among a series of frames. The first time difference 118(1) / 218(1) can be referred to herein as the current frame time and can be derived from the current refresh rate of the display. In an illustrative example, a first time difference 118(1) / 218(1) (or current frame time) of 8.3 ms is considered, which represents an instantaneous refresh rate of 120 Hz. Determining the first time difference 118(1) / 218(1) between a consecutive pair of light pulses 108 / 208 corresponding to a consecutive pair of frames 302 can include determining the time difference between any two corresponding points between a pair of light pulses 108 / 208 or between a pair of frames. For example, the time differences 118 / 218 can be determined between the start point, the mid - point, and the end point of each light pulse 108 / 208. The logic of the display system, in some embodiments, can include frame - start markers in the pixel data between frames rendered by each application, and these frame - start markers can be used to determine the first time difference 118(1) / 218(1) in sub - block 508.

[0071] In sub-block 510, the logic of the display system (e.g., HMD 100 / 200) may determine a first value of the light output parameter based at least in part on the first time differences 118(1) / 218(1). As described herein, the value of the light output parameter may correspond to the magnitude of the light pulse 108(1) for the first frame 302(1). Another value of the light output parameter may correspond to the duration of the light pulse 108(1) for the first frame 302(1). In a rolling band type of display driving method, the value of the light output parameter may correspond to several light emitting elements 106 / 206 (e.g., the thickness of the rolling band of illumination) that are illuminated simultaneously during the light pulse 108(1). In some embodiments, a plurality of these light output parameters may be determined and applied in combination to control the light output.

[0072] In sub-block 512, as part of determining a first value of the light output parameter, the logic may determine a ratio of the first time difference 118(1) / 218(1) to the reference frame time 112 / 212. As described herein, the reference frame time 112 / 212 may be the time corresponding to the intermediate frame rate between the minimum frame rate and the maximum frame rate of the range of frame rates targeted by application 304. For example, if application 304 targets a range of frame rates from 45 FPS to 144 FPS, the reference frame time used to determine the aforementioned ratio may correspond to the frame time at a frame rate of 90 FPS, which is approximately 11.11 ms. In this embodiment, the reference frame time 112 / 212 will be 11.11 ms. However, this is merely one example, and the reference frame time 112 / 212 may be configured based in part on the specifications of the display system in which the techniques described herein are implemented. In the illustrative embodiment, if the first time difference 118(1) / 218(1) (or the current frame time) is 8.3 ms and the reference frame time 112 / 212 is 11.11 ms, the ratio determined in sub-block 512 will be approximately 0.75. This can be considered a light output factor that is 75% of the nominal light output at the reference frame rate 112 / 212, and the light output factor means reducing the light output in some way (whether it be a reduction in the magnitude or duration of light pulse 108(1), etc.). In any case, the first value of the light output parameter may be determined based at least in part on this ratio.

[0073] In sub-block 514, the light-emitting element 106 / 206 can be controlled to emit a light pulse 108(1) (or illumination) according to a first value of the light output parameter during the presentation of the first image on the display. This process 500 can be repeated over a series of frames by dynamically determining the value of the light output parameter for each frame 302 of the series of frames. As a result of varying the light output parameter over the series of frames 302 in this way, the brightness of the display remains substantially constant over the series of frames, thereby eliminating or at least reducing the flicker of the display.

[0074] FIG. 6 shows exemplary components of an HMD 600 (or an HMD system including the HMD 600), such as a VR headset, according to embodiments disclosed herein that may be implemented. The HMD 600 may be the same as or similar to the HMD 100 / 200 referenced in the previous figures, and thus, the components of the HMD 600 shown in FIG. 6 may be implemented in the HMD 100 / 200. The HMD 600 may be implemented as a stand-alone device that will be worn by the user 102 (e.g., on top of the user 102's head). In some embodiments, the HMD 600 may be head mountable, such as by using a fixation mechanism (e.g., an adjustable band) sized to fit around the user 102's head to enable the HMD 600 to be fixed on top of the user's head. In some embodiments, the HMD 600 includes a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display. Thus, the terms “wearable device,” “wearable electronic device,” “VR headset,” “AR headset,” and “head-mounted display (HMD)” may be used interchangeably herein to refer to the device 600 of FIG. 6. However, it should be understood that these types of devices are merely examples of the HMD 600, and it should be understood that the HMD 600 may be implemented in various other form factors. It should also be understood that some or all of the components shown in FIG. 6 may be implemented on the HMD 600. Thus, in some embodiments, a subset of the components shown in FIG. 6 is part of an HMD system but may be implemented on a computing device separate from the HMD 600 itself, such as a PC, a game console, or any other suitable computing device. It should also be understood that a non-HMD system may include some similar components in order to at least implement the techniques and processes of dynamic light output adjustment described herein.

[0075] In the implementation examples described, the HMD 600 includes one or more processors 602 and a memory 604 (e.g., computer-readable medium 604). In some implementation examples, the processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU) 603, both the CPU and the GPU 603, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively, or additionally, what is functionally described herein may be at least partially implemented by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on chip systems (SOC), complex programmable logic devices (CPLD), and the like. Additionally, each of the processors 602 may have its own local memory that can also store program modules, program data, and / or one or more operating systems.

[0076] Memory 604 can include volatile and non-volatile memory, removable media, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such memory can include RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, RAID storage system, or any other medium that can be used to store desired information and can be accessed from a computing device, but is not limited to these. Memory 604 can be implemented as a computer-readable storage medium ("CRSM"), which can be any available physical medium accessible by processor 602 to execute the instructions stored in memory 602. In one basic implementation example, the CRSM can include random access memory ("RAM") and flash memory. In other implementation examples, the CRSM can include read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and can be accessed by processor 602, but is not limited to these.

[0077] Generally, the HMD 600 (or HMD system) may include logic (such as software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 604 is shown as including various modules such as instructions, data stores, etc., and these modules may be configured to execute on the processor 602 to perform the techniques, functions, and / or operations described herein. Although several exemplary functional modules are shown as being stored on the computer-readable medium 604 and executable on the processor 602, the same functions may alternatively be implemented in hardware, firmware, or as system-on-chip (SOC) and / or other logic.

[0078] The operating system module 606 may be configured to manage the hardware coupled to the HMD 600 within the HMD 600 for the convenience of other modules. Additionally, in some instances, the HMD 600 may include one or more applications 304 stored in the memory 604 or otherwise accessible to the HMD 600. In this implementation example, the application 304 includes the game application 608. However, the HMD 600 may include any number or type of applications and is not limited to the specific example shown here. The game application 608 may be configured to initiate gameplay of a video-based interactive game (e.g., a VR game) playable by the user 102 and output frames 302 to be rendered on the display panel of the HMD 600. The compositor 308 may be configured to implement the techniques described herein in combination with other logic of the HMD 600 to render the frames 302 by reprojection, such as by implementation of the process 400. The light output adjustment component 610 may be configured to implement the techniques described herein in combination with other logic of the HMD 600 to dynamically adjust the light output of the light emitting elements 106 / 206 based on the difference in time between consecutive light pulses 108 / 208 corresponding to consecutive frames 302.

[0079] Generally, the HMD500 has an input device 612 and an output device 614. The input device 612 may include control buttons. In some implementations, one or more microphones may function as the input device 612 for receiving audio inputs such as user voice inputs. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function as the input device 612 for receiving gesture inputs such as the movements of the user 102's hand and / or head. In some embodiments, additional input devices 612 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, handheld controller, etc. In other embodiments, the HMD600 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD600 may be implemented with a relatively simple form of the input device 612, a network interface (wireless or wired-based), a power supply, and processing / memory functions. For example, a limited set of one or more input components (e.g., dedicated buttons for starting the configuration, turning the power on / off, etc.) may be used, and the HMD600 may be used thereafter. In one implementation, the input device 612 may include basic volume control buttons for increasing / decreasing the volume, as well as control mechanisms such as a power and reset button.

[0080] The output device 614 may include a display 616, and the display 616 may include one or more display panels 104 / 204 (e.g., a stereo pair of display panels 104 / 204) including the light-emitting elements 106 / 206 as described herein. The output device 614 may further include, but is not limited to, light elements (e.g., LEDs), a vibrator for creating a tactile sensation, speakers (e.g., headphones), etc. For example, there may also be a simple light element (e.g., an LED) for indicating a state such as when the power is on.

[0081] The HMD600 may further include a wireless unit 618 connected to an antenna 620 to facilitate a wireless connection to a network. The wireless unit 618 may implement one or more of various wireless technologies such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the HMD600 may further include a physical port to communicate with a network, connected peripheral devices (including a PC, a game console, etc.), or another wireless network, and to facilitate a wired connection to a plug-in network device that may be part of the HMD system.

[0082] The HMD600 may further include an optical subsystem 622 that uses one or more optical elements to direct light from the electronic display 616 to the user's eyes. The optical subsystem 622 may include various types and combinations of different optical elements, including, but not limited to, an aperture, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and the like. In some embodiments, one or more optical elements within the optical subsystem 622 may have one or more coatings, such as an anti-reflective coating. By magnifying the image light by the optical subsystem 622, the electronic display 616 can be made physically smaller, lighter, and less power-consuming than a larger display. Additionally, the magnification of the image light may increase the field of view (FOV) of the displayed content (e.g., an image). For example, the FOV of the displayed content may be such that the displayed content is presented using substantially all of the user's FOV (e.g., 120 - 150 degrees diagonal), and in some cases, all of it. An AR application may have a narrower FOV (e.g., an FOV of about 40 degrees). The optical subsystem 622 may be designed to correct one or more optical errors, including, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma aberration, field curvature, and astigmatism. In some embodiments, the content provided to the electronic display 616 for display is pre-distorted, and the optical subsystem 622 corrects the distortion when receiving the image light from the electronic display 616 generated based on the content.

[0083] The HMD600 may further include one or more sensors 624, such as sensors used to generate motion, position, and orientation data. These sensors 624 can be, or include, gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors 624 can also include secondary portions of the sensors, such as a series of active or passive markers that can be visually recognized from the outside by a camera or color sensor to generate motion, position, and orientation data. For example, a VR headset may include multiple markers, such as reflectors or lights (e.g., infrared or visible light), that provide one or more reference points for software interpretation to generate motion, position, and orientation data when viewed by an external camera or illuminated by light (e.g., infrared or visible light). The HMD600 may include a light sensor that is sensitive to light (e.g., infrared or visible light) projected or scattered by a base station within the environment of the HMD600.

[0084] In one embodiment, sensor 624 may include an inertial measurement unit (IMU) 626. The IMU 626 may be an electronic device that generates calibration data based on measurement signals received from an accelerometer, a gyroscope, a magnetometer, and / or other sensors suitable for motion detection, correction of errors associated with the IMU 626, or some combination thereof. Based on the measurement signals, a motion-based sensor such as the IMU 626 may generate calibration data indicating the estimated position of the HMD 600 relative to its initial position. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). The IMU 626 may, for example, quickly sample the measurement signals and calculate the estimated position of the HMD 600 from the sampled data. For example, the IMU 626 may integrate the measurement signals received from the accelerometer over time to estimate a velocity vector, and integrate the velocity vector over time to determine the estimated position of a reference point on the HMD 600. The reference point is a point that can be used to describe the position of the HMD 600. The reference point may generally be defined as a point in space, but in various embodiments, the reference point is defined as a point within the HMD 600 (e.g., the center of the IMU 626). Alternatively, the IMU 626 may provide the sampled measurement signals to an external console (or other computing device), and the external console determines the calibration data.

[0085] Sensor 624 may operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data may be generated at a rate of 1000 Hz (or one sensor reading every millisecond). In this way, 1000 readings are taken per second. If the sensor generates this much data at this rate (or at a higher rate), the dataset used to predict motion will be very large even for a relatively short period of a few tens of milliseconds.

[0086] As mentioned, in some embodiments, sensor 624 may include a light sensor that is sensitive to light emitted by a base station within the environment of HMD 600 for the purpose of tracking the position and / or orientation, pose, etc. of HMD 600 in 3D space. The calculation of the position and / or orientation may be based on the timing characteristics of the light pulses and the presence or absence of light detected by sensor 624.

[0087] The HMD600 may further include an eye tracking system 628 that generates eye tracking data. The eye tracking system 628 may include, but is not limited to, a camera or other optical sensor inside the HMD600 to capture image data (or information) of the user's eyes. The eye tracking system 628 can use the captured data / information to determine movement vectors, including the magnitude of twist and rotation (i.e., roll, pitch, and yaw) and the direction of the line of sight of each eye, interpupillary distance, interocular distance, and the three-dimensional (3D) position of each eye relative to the HMD600. In one embodiment, infrared light is emitted within the HMD600 and reflected from each eye. The reflected light is received or detected by the camera of the eye tracking system 628 and analyzed to extract the rotation of the eyes from the changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 102 can be used by the eye tracking system 628. Thus, the eye tracking system 628 can track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) for each eye, and at least a subset of the tracked amounts can be combined from the two eyes of the user 102 to estimate the point of the line of sight (i.e., the 3D location or position within the virtual scene that the user is looking at). The point of the line of sight can be mapped to a location on the display panel 104 / 204 to predict where the user 102 is looking with respect to individual subsets (e.g., rows) or groups of contiguous subsets (e.g., groups of contiguous rows) of pixels on the display panel 104 / 204. For example, the eye tracking system 628 can integrate information from past measurements, measurements that identify the position of the head of the user 102, and 3D information that describes the scene presented by the electronic display 616. Thus, information about the position and orientation of the eyes of the user 102 is used to determine the point of the line of sight within the virtual scene presented by the HMD600 that the user 102 is looking at and to map that point of the line of sight to a location on the display panel 104 / 204 of the HMD600. The movement vectors can be utilized to predict, for example, the trajectory of the eye movement. It can be utilized for prediction.

[0088] The HMD 600 may further include a head tracking system 630. As described above, the head tracking system 630 may utilize one or more of the sensors 624 to track head movements, including rotations of the user 102's head. For example, the head tracking system 630 may track up to six degrees of freedom of the HMD 600 (i.e., 3D position, roll, pitch, and yaw). These calculations can be performed for all of the frames 302 of a sequence of frames 302, such that the application 304 can determine how to render the scene within the next frame 302 based on the head position and orientation, and such that the compositor 308 can perform computer processing for reprojection adjustment. In some embodiments, the head tracking system 630, and / or the compositor 308 that uses the head tracking system 630, is configured to predict the future pose (position and / or orientation) of the HMD 600 based on current and / or past data, and / or based on the known / implied scan-out latency of individual subsets of pixels within a display system that implements a rolling band display driving technique, as described herein. This is because the application 304 is required to render the frame 302 before the user 102 actually views the light (and, by extension, the image) on the display 616. Thus, the next frame 302 can be rendered based on a future prediction of the head position and / or orientation made at an earlier point in time that may be within the range of approximately 12 to 30 milliseconds (ms) before rendering the frame 302, depending on the instantaneous frame rate and / or refresh rate. The rotational data provided by the head tracking system 630 can be used to determine both the direction of rotation and the amount of rotation of the HMD 600 in any suitable unit of measurement. For example, the direction of rotation can be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation can be expressed in degrees, radians, etc. The angular velocity can be calculated to determine the rate of rotation of the HMD 600.

[0089] Although the subject matter has been described in language specific to structural features, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as exemplary forms of implementing the claims.

Claims

1. 1. A display system comprising: a display having an array of light emitting elements, the display being configured to support a variable refresh rate; one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the display system to: outputting first pixel data associated with a first frame of the sequence of frames to a frame buffer; determining a first time difference between illumination of the light emitting element for a previous frame of the series of frames and a next illumination of the light emitting element for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and illuminating the light emitting elements in accordance with the first value of the light output parameter during presentation of the first image on the display.

2. The computer-executable instructions, when executed by the one or more processors, cause the display system to: outputting second pixel data associated with a second frame of the series of frames to the frame buffer; and determining a second time difference between an illumination of the light emitting element for the first frame and a subsequent illumination of the light emitting element for the second frame; determining a second value of the light output parameter based at least in part on the second time difference; and presenting a second image on the display based at least in part on the second pixel data; During presentation of the second image on the display, and illuminating the light emitting elements according to a value of 2; the second time difference is greater than the first time difference; the second value of the light output parameter causes the light emitting element to emit light at a greater brightness or for a longer duration than the first value of the light output parameter; the second time difference is less than the first time difference; 2. The display system of claim 1, wherein the second value of the light output parameter causes the light emitting elements to emit light at a brightness less than or for a duration shorter than the first value of the light output parameter.

3. The first value of the light output parameter is: the magnitude of a pulse of light for illuminating said light-emitting element; the duration of a pulse of the light to illuminate the light-emitting element; or The display system of claim 1 , wherein the sequential illumination of individual subsets of the light-emitting elements during the presentation of the first image on the display corresponds to at least one of the number of the light-emitting elements that are simultaneously illuminated during the sequential illumination of individual subsets of the light-emitting elements.

4. determining the first value of the light output parameter based at least in part on the first time difference; determining a ratio of the first time difference to a reference frame time; and determining said first value of said light output parameter based at least in part on said ratio.

5. 5. The display system of claim 4, wherein the application that rendered the first frame is configured to target a frame rate within a range of frame rates from a minimum frame rate to a maximum frame rate, and the reference frame time corresponds to a frame time corresponding to an intermediate frame rate between the minimum frame rate and the maximum frame rate.

6. and wherein the display is a head mounted display (HMD) and the display system further comprises a head tracking system, and the computer executable instructions, when executed by the one or more processors, prior to outputting the first pixel data to the frame buffer, provide the display system with: determining a predicted illumination time representative of a time that the light emitting element will illuminate for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representative of the time that the light emitting elements will illuminate for the first frame based at least in part on an amount of time the application spent rendering the first frame; a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; and determining 2. The display system of claim 1 , further comprising: applying a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame.

7. determining the predicted illumination time, determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The display system of claim 6 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

8. The computer-executable instructions, when executed by the one or more processors, cause the display system to: determining a second amount of time that the application spent rendering a previous frame that precedes the preceding frame; determining an average rendering time based at least in part on the first amount of time and the second amount of time; The display system of claim 7 , wherein said determining said predicted rendering time is based at least in part on said average rendering time.

9. The display system of claim 1 , wherein the display system is at least one of a virtual reality (VR) display system or an augmented reality (AR) display system.

10. 1. A method implemented by a display system including a display having an array of light emitting elements, the method comprising: outputting first pixel data to a frame buffer, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between a light pulse of the light emitting element for a previous frame of the series of frames and a next light pulse of the light emitting element for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and controlling the light emitting elements to emit light pulses according to the first value of the light output parameter during presentation of the first image on the display. method.

11. The first value of the light output parameter is: the magnitude of the write pulse for the first frame; the duration of the write pulse for the first frame, or The method of claim 10 , wherein the at least one of the number of light emitting elements simultaneously illuminated during the light pulse corresponds to at least one of the number of light emitting elements simultaneously illuminated during the light pulse.

12. determining the first value of the light output parameter based at least in part on the first time difference; determining a ratio of the first time difference to a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

13. the display is a head mounted display (HMD), and the method further comprises, prior to outputting the first pixel data to the frame buffer, determining an expected illumination time representative of a time when the light pulse will occur for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by a head tracking system of the display system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time representing the time at which the light pulse for the first frame will occur based at least in part on an amount of time the application spent rendering the first frame; determining a new predicted pose that the HMD will assume at the new predicted illumination time based at least in part on the head tracking data generated by the head tracking system; 11. The method of claim 10, further comprising: applying a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame.

14. determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The method of claim 13 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

15. said controlling said light emitting elements to emit said light pulses causes said light emitting elements to emit light simultaneously; or The method of claim 10 comprising at least one of: causing individual subsets of the light emitting elements to sequentially emit the light.

16. 1. A system comprising: a display having an array of light sources; a display driver circuit coupled to the array of light sources; one or more processors, the one or more processors: outputting first pixel data to a frame buffer, the first pixel data being associated with a first frame of a series of frames; determining a first time difference between an illumination of the light source for a previous frame of the series of frames and a next illumination of the light source for the first frame; determining a first value of a light output parameter based at least in part on the first time difference; presenting a first image on the display based at least in part on the first pixel data; and causing the display driver circuit to illuminate the light source according to the first value of the light output parameter during presentation of the first image on the display.

17. The first value of the light output parameter is: the magnitude of a pulse of light for illuminating said light source; the duration of the light pulse for illuminating the light source; or 17. The system of claim 16, wherein the sequential illumination of individual subsets of the light sources during the presentation of the first image on the display corresponds to at least one of a number of the light sources simultaneously illuminated.

18. determining the first value of the light output parameter based at least in part on the first time difference; determining a ratio of the first time difference to a reference frame time; and determining the first value of the light output parameter based at least in part on the ratio.

19. The method further comprises the step of: providing a head tracking system, the display being a head mounted display (HMD), and the one or more processors, before outputting the first pixel data to the frame buffer, determining a predicted illumination time representative of a time that the light source will illuminate for the first frame; determining a predicted pose that the HMD will assume at the predicted illumination time based at least in part on head tracking data generated by the head tracking system; sending pose data indicative of the predicted pose to an application for rendering the first frame; receiving pixel data for the first frame from the application; determining a new predicted illumination time that represents the time that the light source will illuminate for the first frame based at least in part on an amount of time the application spent rendering the first frame; At least in part, the head tracking data generated by the head tracking system determining a new predicted pose that the HMD will assume at the new predicted illumination time based on 17. The system of claim 16, further configured to: apply a reprojection adjustment to the pixel data based at least in part on the predicted pose and the new predicted pose to obtain the first pixel data associated with the first frame.

20. the one or more processors: determining a first amount of time that the application spent rendering the previous frame; determining a predicted rendering time representing an amount of time the application will spend rendering the first frame based at least in part on the first amount of time; The system of claim 19 , wherein the predicted illumination time is determined based at least in part on the predicted rendering time.

Images