Dynamic Panel Masking

Abstract

The panel mask(s) rendered on the display panel(s) of a head-mounted display (HMD) dynamically adjust (increase and decrease) in size as needed to hide undesired visual artifacts from view. For example, when frames are being rendered on the display panel of the HMD using reprojection, a size value associated with at least a portion of the panel mask is adjusted based on rotation of the HMD, increasing or decreasing the size of at least a portion of the panel mask from its current size to an adjusted size, such that at least a portion of the panel mask is rendered at the adjusted size to hide undesired visual artifacts. The size of the portion of the panel mask is subsequently decreased over a period of time when reprojection stops and / or when head rotation stops or slows. [Selected Figure] Figure 1A

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This is a PCT application claiming priority to U.S. Patent Application No. 16 / 033,162, entitled "Dynamic Panel Masking," filed July 11, 2018, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Virtual reality (VR) systems are used both within and outside the video game industry. Displays for VR systems, such as those built into VR headsets, typically operate at a minimum refresh rate that is suitable for VR applications. For example, 90 Hertz (Hz) is a common refresh rate for VR displays. In a "live rendering" scenario, a graphics-based application, such as a video game, outputs frames for rendering at a frame rate that matches the refresh rate of the display. That is, new frames received from the application (referred to herein as "actual frames") are displayed each time the screen is updated. Such live rendering scenarios are often referred to as the application "reaching frame rate."

[0003] In reality, applications may not always reach frame rate for a variety of reasons. For example, an application may intermittently drop frames and / or may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). In situations where an application is not reaching frame rate, a technique called "rotation-only reprojection" can be used to replace missing frames with reprojected frames that account for the user's head rotation, thereby making the application appear to the user as if it is reaching frame rate. For example, without reprojection, a lack of frame rate from an application may result in stuttering or hitching in a game. In VR applications where the user is fully immersed in a virtual environment, if the application does not reach frame rate and there is no reprojection to make up for the missed frames, the user may experience discomfort. Therefore, reprojection is a technique that improves the user experience when an application is not reaching frame rate. Consider an example where an application is outputting frames at half the ideal frame rate (e.g., 45 frames per second when 90 frames per second is the ideal frame rate). In this example, every other frame can be reprojected by using pixel data from the most recently rendered actual frame to create a reprojected frame that transforms the scene (e.g., through rotation and reprojection calculations) and adapts the reprojected scene to the user's current head orientation. This allows the scene to appear to the user to move predictably as the user's head rotates, even when reprojected frames are used to compensate for applications that cannot reach frame rates.

[0004] While rotation-only reprojection prevents in-game stutter or hitches, it creates its own undesirable visual artifacts during head rotation, at least in VR systems using low-persistence displays (e.g., when the display is illuminated for only a small fraction of the frame time). For example, with rotation-only reprojection, a strobe flash can appear at the leading edge of each display panel in the direction of head rotation as the user rotates their head. This strobe flash occurs during head rotation because the pixels at the leading edge of the display panel switch every frame between valid pixels in the actual frame and solid black pixels in the next reprojected frame. Black pixels occur in reprojected frames during head rotation because there is no pixel data available from the previously rendered actual frame to use for the pixels at the leading edge of the display panel for the reprojected frame. Thus, when the user rotates their head while reprojection is in use, the leading edge of the display panel in the direction of the user's head rotation switches every frame between black and illuminated pixels, thereby creating undesirable strobe artifacts at the leading edge of the display panel.

[0005] Technical solutions are provided herein to improve and enhance these and other systems. [Brief explanation of the drawings]

[0006] The detailed description will now be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number refers to the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or functions.

[0007] [Figure 1A] FIG. 1 illustrates exemplary left and right display panels of a head-mounted display (HMD), where frames are rendered with dynamically expanded panel masks during head rotation while the frames are processed using reprojection, and the head rotation direction is to the right (or positive X direction). [Figure 1B] FIG. 1B illustrates exemplary left and right display panels of the HMD of FIG. 1A, where frames are rendered with dynamically expanded panel masks during head rotation while the frames are processed using reprojection, and the head rotation direction is to the left (or negative X direction). [Figure 2] 1A-1C are diagrams illustrating example left and right display panels of an HMD showing the difference between a fully contracted state of the panel mask and a fully expanded state of the panel mask. [Figure 3] FIG. 10 is another diagram illustrating example left and right display panels of an HMD showing the difference between a fully contracted state of the panel mask and a fully expanded state of the panel mask. [Figure 4] FIG. 10 is a flow diagram of an example process for dynamically resizing a portion of a panel mask on a display panel of an HMD, according to embodiments disclosed herein. [Figure 5] FIG. 10 is a flow diagram of a more detailed exemplary process for dynamically resizing a portion of a panel mask on a display panel of an HMD, according to embodiments disclosed herein. [Figure 6] 1 illustrates exemplary components of a wearable device, such as a VR headset, in which the techniques disclosed herein can be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0008] Described herein, among other things, is a technique for dynamically adjusting (increasing and decreasing) the size of a panel mask rendered around the periphery of a display panel of a head-mounted display (HMD) to hide undesirable visual artifacts from view as needed. The HMD may be worn by a user for the purpose of immersing the user in a virtual reality (VR) or augmented reality (AR) environment. One or more display panels of the HMD render images based on frames output by an application (e.g., a video game), and these images are displayed to the user through optics included in the HMD, causing the user to perceive the images as if the user were immersed in the VR or AR environment.

[0009] The panel mask(s) may be rendered around the periphery of the HMD's display panel(s) to provide a boundary around the image presented on the display panel(s). The panel mask frames the image much like a picture frame. In some embodiments, the inner edges of the panel mask(s) may be blurred (e.g., using a subtle decorative pattern) to make the panel mask appear to the user as a near-field object close to (or disposed on) the user's face, closely resembling facial features seen when focusing on a far-field object (e.g., the bridge of the nose between the eyes).

[0010] As mentioned, HMDs can compensate for applications that do not reach the frame rate using a technique called “reprojection.” During reprojection, a user wearing an HMD may rotate their head, which, as mentioned, can cause undesirable strobe artifacts to appear at the leading edge of the display panel(s) not currently covered by the panel mask. Therefore, the HMD can detect whether reprojection is being used, and at least a portion of the panel mask can dynamically adjust its size based on whether reprojection is being used. For example, when reprojection is in progress, the portion of the panel mask can be adjusted by an appropriate amount (e.g., shrinking away from the center of the display panel or expanding toward the center of the display panel) depending on the amount of rotation of the HMD during reprojection. This can mask (or hide) undesirable visual artifacts, such as strobe flashes, that may appear during reprojection in conjunction with head movement. For example, if a user rotates their head to the right while frames are being reprojected, the size of the portion of the panel mask at the front (e.g., right) edge of the display panel(s) may be increased to ensure that the panel mask covers any undesired strobe artifacts, thereby preventing the user from seeing strobe flashes in the image area of ​​the display panel. Subsequently, if the user slows or stops their head rotation, or if the application reaches frame rate again (i.e., stops using reprojection), the size of this expanded portion of the panel mask may be adjusted over time back to its fully contracted state, thereby allowing a larger scene to be rendered in the image area of ​​the display panel. This technique of dynamically expanding and contracting at least a portion of the panel mask based on a combination of head rotation and reprojection allows a better viewing experience because the user of the HMD does not notice the undesired strobe artifacts caused by reprojection during head rotation. The panel mask expands to cover the strobe artifacts when they are present, and contracts when the strobe artifacts are no longer present, revealing a larger scene in the image area of ​​the display panel.

[0011] An exemplary process for dynamically adjusting a size of a portion of a panel mask according to embodiments described herein includes: determining, by one or more processors, whether a series of frames are being rendered on a display panel of an HMD using reprojection; determining a target size value for at least a portion of the panel mask based at least in part on whether the series of frames are being rendered using reprojection; determining that a size value associated with at least a portion of the panel mask is set to an existing size value that is not equal to the target size value; adjusting the size value from the existing size value to the target size value to increase or decrease a size of at least a portion of the panel mask from a current size to an adjusted size; and rendering a frame of the series of frames on the display panel with the panel mask rendered around the periphery of the display panel, wherein at least a portion of the panel mask has been rendered at a size adjusted according to the target size value. If the size adjustment is to increase the size of the portion of the panel mask, the size of the portion of the panel mask may subsequently be decreased in size when the use of reprojection stops and / or head rotation slows or stops completely. For example, when reprojection is no longer in use, the size of the portion of the panel mask may be gradually reduced over a period of time to a minimum size or other target value, so that the reduction of the panel mask back to a fully reduced state is not visually distracting to the user.

[0012] Also disclosed herein are systems including HMDs configured to implement the techniques and processes disclosed herein, and non-transitory computer-readable media storing computer-executable instructions for implementing the techniques and processes disclosed herein. While the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, particularly VR game applications, it should be understood that the techniques and systems described herein may also provide benefits to other applications in which HMDs are used, including, but not limited to, non-VR applications (e.g., AR applications) and / or non-game applications such as industrial machinery applications, defense applications, robotics applications, etc.

[0013] 1A is a diagram illustrating an exemplary head-mounted display (HMD) 100 including a left display panel 102(L) and a right display panel 102(R) that form a pair of display panels. While the exemplary HMD 100 includes two display panels 102(L) and 102(R), it should be understood that the HMD 100 may include a single display panel 102, or three or more display panels 102. Accordingly, a "display panel 102," as used herein, may refer to either the display panel 102(L) or 102(R) of a two-panel HMD 100, as shown in FIG. 1A, or may refer to a single display panel 102 of an HMD 100 with any number of display panels (e.g., a single-panel HMD 100 or a multi-panel HMD 100).

[0014] The display panel(s) 102 may represent any suitable type of display, such as a light-emitting display that utilizes light-emitting elements to emit light during the presentation of an image frame (referred to herein as a "frame") on the display panel(s) 102. By way of example, the left and right display panels 102(L) and 102(R) may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display that utilizes a display technology suitable for HMD applications.

[0015] The HMD 100 may represent a VR headset for use in a VR system, such as for use in a VR gaming system. However, the HMD 100 may additionally or alternatively be implemented as an AR headset for use in an AR application. In AR, a user sees virtual objects superimposed on a real-world environment, whereas in VR, a user does not see a real-world environment but is fully immersed in a virtual environment as perceived through the display panel 102 and optical system (e.g., lenses) of the HMD 100. While the examples described herein primarily relate to a VR-based HMD 100, it should be understood that the HMD 100 is not limited to implementation in VR applications.

[0016] In general, an application running on a computing device such as the HMD 100 itself, or a computing device associated with and coupled to the HMD 100 (e.g., a personal computer (PC), game console, etc.), may be configured to output a series of frames 106 that are ultimately rendered on the display panel(s) 102 of the HMD 100. Image data for the frames 106 is presented within image regions 108 of the display panel(s) 102 that are not covered by a panel mask 110. FIG. 1A shows a left image region 108(L) that is centered on the left display panel 102(L) and inside the left panel mask 110(L), and a right image region 108(R) that is centered on the right display panel 102(R) and inside the right panel mask 110(R).

[0017] The series of frames 106 may be output from a video game application or any other type of graphics-based application. The application may execute in a graphics pipeline that outputs per-pixel values ​​(e.g., color values) that are combined to create an image (in image region 108) on display panel(s) 102 with a desired visual effect. At runtime, the head tracking module of HMD 100 may generate data regarding the position and orientation of HMD 100, which is provided to the application to inform the application on how to render the next frame 106 in the series of frames 106 depending on the head position / orientation of user 104. This allows images to be rendered in image region(s) 108 on display panel(s) 102 in a way that makes the user 104 believe they are looking around a virtual environment, including objects (both static and moving), which move predictably through the scene as the user 104 moves their head.

[0018] As mentioned, the left panel mask 110(L) is rendered around the periphery of the left display panel 102(L), and the right panel mask 110(R) is rendered around the periphery of the right display panel 102(R). As used herein, "panel mask 110" may refer to either the panel mask 110(L) or 110(R), as shown in FIG. 1A . However, a single-panel HMD 100 may include a single panel mask 110 that surrounds the image region 108 of a single display panel 102. At startup, the HMD 100 may calculate a mesh for the panel mask(s) 110, which is used to render the panel mask(s) 110 at full scale around the periphery of the display panel(s) 102, meaning that portions of the panel mask(s) 110, whose size is dynamically adjustable, are initially rendered at a minimum size to display a larger scene within the image region(s) 108. Generally, panel mask(s) 110 may be rendered continuously with each frame, although the size of individual portions of panel mask(s) 110 may be adjusted, if necessary, in particular ones of the frames to hide undesirable visual artifacts. In some embodiments, panel mask(s) 110 may be implemented as a plurality of black pixels at the periphery of display panel(s) 102, which may be added to a frame buffer before rendering a frame. While panel mask(s) 110 are illustrated in the figures as approximately annular panel masks surrounding approximately circular image region 108, it should be understood that other shapes may be used to implement panel mask(s) 110, such as rendering panel mask(s) 110 as a square, rectangular, or other geometrically shaped frame surrounding image region 108. In some embodiments, the panel mask(s) 110 may be implemented as three separate panel masks corresponding to the red, green, and blue color channels, with each separate panel mask 110 of the three panel masks 110 shaped based on known color distortions of the optics included in the HMD 100 to mitigate the effects of color fringing that may occur at or near the outer edges of the display panel 102.For example, when the user 104 looks through the lenses in the HMD 100, there may be some natural spatial (radial) distortion due to the lenses distorting the image to some extent (known as the "pincushion" effect in VR headsets). There may also be some color distortion (e.g., red, green, and blue (RGB) separation). If the user 104's eyes happen to be close enough to the lens assembly so that the user 104 can actually see the edges of the display panel 102, the user 102 may see a cyan fringe on the edge of the panel due to the color distortion separating RGB, with the red pixels (toward the center of the display panel 102) being the most chromatically distorted, followed by the green pixels less distorted, and the blue pixels least distorted (e.g., blue is the least distorted and remains closest to the edge of the panel, causing the cyan fringe effect). This color fringing can be very distracting to a user who may be able to see it (the color fringing may or may not be visible to the user 104 depending on the thickness of the foam used in the HMD 100, the spatial shape of the user's face, etc.). Thus, separate panel masks of a three-panel mask implementation on each display panel 102 can independently mask red, green, and blue to mitigate the color fringing effect.

[0019] 1A illustrates the left panel mask 110(L) as a relatively dark gray area and the right panel mask 110(R) as a relatively light gray area. This is merely to distinguish the two panel masks from each other in the relevant figures, and it should be understood that the pair of panel masks 110(L) and 110(R) may be rendered in a uniform color (e.g., black pixels) on both display panels 102(L) and 102(R). In this manner, the user 104 perceives the two panel masks 110(L) and 110(R) as a single panel mask surrounding the two image regions 108(L) and 108(R), similar to how a human perceives the environment with the eye sockets and nose bridge surrounding each eyeball at close range. As mentioned, the inner edge 112 of each panel mask 110(L) and 110(R) may be slightly blurred (e.g., using a subtle decorative pattern) to resemble near-field facial features more familiar to humans than sharp inner edges 112.

[0020] Additionally, the size of the individual portions of each of the panel masks 110(L) and 110(R) is dynamically adjustable, expanding and contracting the portions of the panel mask 110 as needed to cover more or less of the image area 108 for each frame 106 in the series of frames 106. For example, the size of a portion of the panel mask(s) 110 can be increased by expanding it inward toward the center of the display panel(s) 102, thereby covering (or hiding) undesirable visual artifacts that appear in that portion of the display panel(s) 102 during reprojection in conjunction with head rotation.

[0021] As mentioned, displays used in VR systems typically operate at a minimum refresh rate suitable for VR applications. Thus, the display panel(s) 102 of the HMD 100 may operate at a refresh rate of, for example, 90 Hz, which is a common refresh rate for VR displays. While 90 Hz is used as an exemplary refresh rate, it should be understood that the display panel(s) 102 may operate at other different refresh rates without changing the fundamental characteristics of the techniques and systems disclosed herein. In a "live rendering" scenario, an application outputting a series of frames 106 may output the actual frames 106(A) at a frame rate that matches the refresh rate of the display panel(s) 102 (e.g., the actual frames 106(A) are output at 90 frames per second). However, if the application is not reaching the frame rate (i.e., outputting actual frames 106(A) at less than 90 frames per second or so), the HMD 100 may be configured to employ reprojection (e.g., rotation-only reprojection) to replace missing actual frames 106(A) with reprojected frames 106(R), and the HMD 100 may do so in a manner that accounts for the head rotation of the user 104, so that the scene appears to the user 104 to move around the display panel 102 as expected if the application were reaching the frame rate and the user 104's head were to rotate. FIG. 1A illustrates this reprojection scenario, where a series of frames 106 includes an actual frame 106(A)(1), followed by a reprojected frame 106(R)(2), followed by another actual frame 106(A)(3), etc. For example, in the example of Figure 1A, the application may output frames at half the ideal frame rate (e.g., at a frame rate of 45 frames / second if 90 frames / second is the ideal frame rate), where every other frame between successive actual frames 106(A) is a reprojected frame 106(R).Reprojected frame 106(R) is generated using pixel data from a recently rendered actual frame 106(A) (e.g., the most recently rendered actual frame 106(A)) to create a scene that is transformed (e.g., through rotation and reprojection calculations) in a manner that accounts for the head rotation of user 104. Assuming actual frame 106(A)(1) is rendered before reprojected frame 106(R)(2) of FIG. 1A , reprojected frame 106(R)(2) can be derived from pixel data associated with previously rendered actual frame 106(A)(1). In this case, the scene in previously rendered actual frame 106(A)(1) is rotated and reprojected to create reprojected frame 106(R)(2) as user 104 would expect if they moved their head.

[0022] As a series of frames 106 are being rendered onto the display panel(s) 102 of the HMD 100 using reprojection, and as the user 104 rotates their head, strobe flashes may appear at the leading edge of each display panel 102 in the direction of head rotation. For example, in FIG. 1A , the user 104 is shown rotating their head in a right (i.e., positive X) rotation direction. In this scenario, the aforementioned strobe flashes may occur during reprojection and may appear at the right edge (i.e., leading edge in the direction of rotation) of each display panel 102(L) and 102(R) because, as the user 106 has rotated to the right, there is no pixel data available in the previously rendered actual frame 106(A)(1) to render pixels at the right edge of each display panel 102(L) and 102(R) for the reprojected frame 106(R)(2). Therefore, the HMD 100 may be configured to dynamically increase the size of the right portion of each panel mask 110 to cover (or hide) this undesirable visual artifact. Without changing the size of the right portion of each panel mask 110, this strobe artifact may be visible in the image region(s) 108 of the display panel(s) 102.

[0023] In illustrative examples, the HMD 100 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to determine whether a series of frames 106 are being rendered on the display panel(s) 102 of the HMD 100 using reprojection. That is, the logic may determine whether one or more frames 106 in the series of frames 106 are reprojected frames 106(R) generated from pixel data associated with an actual frame 106(A) received from an application that has already been rendered on the display panel(s) 102. In some embodiments, the logic may make this determination when rendering the reprojected frame 106(R)(2). In other embodiments, the logic may make this determination after rendering the actual frame 106(A)(1) but before rendering the reprojected frame 106(R)(2). In any event, if reprojection is used, the logic may also be configured to determine a target size value for each adjustable portion of panel mask(s) 110 based at least in part on rotation data provided by a head tracking module of HMD 100 indicating that HMD 100 has rotated (e.g., between time t1 and time t2). If the target size value for a given portion of panel mask(s) 110 is not equal to the existing size value of that portion, the logic may respond by adjusting the size value associated with at least the portion of panel mask(s) 110 from the existing size value to the target size value, increasing the size of at least the portion of panel mask(s) 110 from its current size to the adjusted size, and rendering a frame (e.g., actual frame 106(A)(3)) of the series of frames 106 on display panel(s) 102 with panel mask(s) 110 rendered around the display panel(s) 102, where at least the portion of panel mask(s) 110 is rendered at the adjusted size.Consider an example in which the left panel mask 110(L) and the right panel mask 110(R) are each divided into left, right, top, and bottom portions. FIG. 1A shows that the size of the right portion of the left panel mask 110(L) and the right portion of the right panel mask 110(R) (each portion corresponds to a right rotation direction) has been increased from its current size to an increased size. This is illustrated in FIG. 1A by the inner edge 112(L) of the left panel mask 110(L) moving inward (i.e., from a dashed line to a solid line) toward the center of the left display panel 102(L), and the inner edge 112(R) of the right panel mask 110(R) moving inward (i.e., from a dashed line to a solid line) toward the center of the right display panel 102(R). In other words, the right portion of each panel mask 110 is pulled inward, increasing the size of the right portion of each panel mask 110, so that a larger portion of the image area 108 on the right edge / right side of the display panel(s) 102 is covered by the panel mask(s) 110. These "inward" adjustments occur when the target size value for that portion of the panel mask(s) 110 is larger than the existing size value.

[0024] 1A , where frame 106 (e.g., actual frame 106(A)(3)) is rendered with dynamically enlarged panel mask(s) 110 during head rotation while frame 106 is processed using reprojection, the direction of head rotation being to the left (or negative X) in FIG. 1B , opposite the direction of rotation shown in FIG. 1A . In the FIG. 1B scenario, the logic of HMD 100 is configured to respond similarly to how it responded in the FIG. 1A scenario, except that a different portion (i.e., the left portion) of each panel mask 110 is resized in response to the different rotation directions detected by HMD 100. For example, if there is a leftward (i.e., negative X) rotation during reprojection, the left portion of each panel mask 110 is pulled inward, increasing the size of the left portion of the panel mask 110, resulting in a larger portion of the image area 108 on the left edge / left side of the display panel(s) 102 being covered by the panel mask(s) 110.

[0025] Similarly, it can be appreciated that an upward or downward rotation of the head can dynamically increase the size of the upper and lower portions of each panel mask 110, respectively (e.g., by expanding inward toward the center of the display panel(s) 102), resulting in larger portions of the image area 108 on the upper edge / upper side and lower edge / lower side of the display panel(s) 102 being covered by the panel mask(s) 110. It can be appreciated that if the rotational orientation of the HMD 100 is a vector having both an X component and a Y component, this can simultaneously expand the size of adjacent portions of the panel mask(s) 110. For example, if the user 104 rotates their head diagonally toward the positive XY quadrant of the reference coordinate system shown in FIGS. 1A and 1B , this can simultaneously expand the size of both the right and upper portions of the panel mask(s) 110, thereby simultaneously expanding the two adjacent portions of the panel mask(s) 110 inward toward the center of the display panel 102. It may also be understood that in some implementations, the entire (i.e., all portions) of the panel mask(s) 110 may be enlarged in size, causing the image area(s) 108 to shrink (or become smaller) as a larger portion of the image area(s) 108 is covered by the enlarged panel mask(s) 110. If the entire panel mask(s) 110 is subsequently shrunk to a smaller size, the image area(s) 108 may expand back to the maximum size of the image area(s) 108 as a smaller portion of the image area(s) 108 is covered by the shrunken panel mask(s) 110.

[0026] FIG. 2 is a diagram illustrating exemplary left and right display panels 102 of an HMD 100, showing the difference between a fully contracted state of the panel mask(s) 110 at the top of FIG. 2 and a fully expanded state of the panel mask(s) 110 at the bottom of FIG. 2. As mentioned, individual portions of the panel mask 110 for each display panel 102 can dynamically adjust in size as needed to cover more or less of the image area 108 of each display panel 102. These size adjustments may be subject to restrictions or limits. For example, individual portions of the panel mask 110 may increase up to, but not exceed, a maximum size, and may decrease down to, but not below, a minimum size. In this sense, the panel mask 110 may be rendered continuously with each frame, with its size controllable based on a value ranging from a minimum size value (e.g., “0”) to a maximum size value (e.g., “1”). In other words, the size values ​​of the individual portions of panel mask 110 are fixed at a minimum size value (e.g., "0") to a maximum size value (e.g., "1"), but are adjustable to any number between and including these two limits. One way to think of this is as a percentage adjustment (e.g., 0 is 0%, 0.4 is 40%, 0.6 is 60%, 1 is 100%, etc.). As noted above, this size adjustment can be controlled independently for each individual portion of panel mask 110.

[0027] For example, consider dividing the left panel mask 110(L) into four portions, including a left portion 110(L)(1), an upper portion 110(L)(2), a right portion 110(L)(3), and a lower portion 110(L)(4). Similarly, the right panel mask 110(R) can be divided into four portions, including a left portion 110(R)(1), an upper portion 110(R)(2), a right portion 110(R)(3), and a lower portion 110(R)(4). Each of these portions of the panel mask 110 can be adjusted in size between a minimum size value (e.g., a size value set to "0") corresponding to a fully contracted state at the top of FIG. 2 and a maximum size value (e.g., a size value set to "1") corresponding to a fully expanded state at the bottom of FIG. 2. Thus, there can be at least four dynamically adjustable size values ​​for each panel mask 110 to control the size of the individual portions of the panel mask 110. The inner edges 112 of the panel masks 110 (at each portion of the panel mask 110) can be moved between a minimum range (shown at the top of FIG. 2) and a maximum range (shown at the bottom of FIG. 2) by setting the size value to a value between the minimum and maximum size values ​​(e.g., a size value between "0" and "1"). There may be reasons to persistently render the panel mask(s) 110 at a minimum size (i.e., fully scaled down), regardless of whether reprojection is used and / or head rotation is present. For example, some HMDs 100 may exhibit color fringing at or near the outer edges of the display panel 102, and it may be desirable to cover (or hide) such visual artifacts so as not to distract the user 104. On the other hand, the panel mask(s) 110 may need to be scaled up beyond the maximum size value because there are practical limitations on the speed at which the HMD 100 can be rotated when worn by the user 104, and there are practical limitations on perceptible visual elements when the user 104 is rapidly rotating their head. For example, it may be virtually impossible for an average user 104 to rotate their head beyond a certain threshold angular velocity, and therefore, at this point, the panel mask 110 does not need to be expanded to cover the entire display panel 102.The expansion of the panel mask can "take in" enough of the field of view to cover the undesirable visual artifacts. Furthermore, even if the user 104 were able to rotate their head beyond this threshold angular velocity, the user 104 would not be able to identify the undesirable visual artifacts that appear with such rapid rotation, because the eyes simply cannot track these visual elements when the field of view is changing rapidly.

[0028] FIG. 3 is another diagram illustrating exemplary left and right display panels 102(L) and 102(R) of the HMD 100, showing the difference between a fully retracted state of the panel masks 110(L) and 110(R) and a fully expanded state of the panel masks 110(L) and 110(R). In FIG. 3, the fully retracted state is indicated by a dashed line representing where the inner edge 112 of the panel mask 110 is in the fully retracted state. The size value (SV) of each portion (e.g., top, bottom, left, and right) of the panel mask 110 can be set to zero (e.g., SV=0) to move the inner edge 112 of that portion of the panel mask 110 outward from the center of the display panel 102, thereby reducing the portion of the panel mask 110 to its minimum size. Individual portions of the panel mask 110 can have their size values ​​set to zero upon power-up of the HMD 100. 3 , the fully expanded state of panel mask(s) 110 is indicated by a solid line representing where inner edge 112 of panel mask 110 is in the fully expanded state. The size value (SV) of each portion of panel mask 110 can be set to 1 (e.g., SV=1) to move inner edge 112 of that portion of panel mask 110 inward toward the center of display panel 102, thereby increasing the portion of panel mask 110 to its maximum size. As mentioned, individual portions of panel mask 110 can be adjusted to intermediate sizes between the minimum and maximum sizes by setting the size value of each portion to an intermediate value between the minimum size value (e.g., “0”) and the maximum size value (e.g., “1”).

[0029] The processes described herein are illustrated as a collection of blocks in a logic flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or combinations thereof (i.e., logic). In the software context, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement a process.

[0030] 4 is a flow diagram of an exemplary process 400 for dynamically resizing a portion of a panel mask 110 on a display panel 102 of an HMD 100, according to an embodiment disclosed herein. For discussion purposes, the process 400 will be described with reference to the previous figure.

[0031] At 402, logic of HMD 100 can determine target size value(s) for at least a portion(s) of panel mask(s) 110. For a given frame 106, logic of HMD 100 can determine, for example, four target size values ​​corresponding to left portion 110(L)(1), top portion 110(L)(2), right portion 110(L)(3), and bottom portion 110(L)(4) of left panel mask 110(L), and similarly, four target size values ​​corresponding to left portion 110(R)(1), top portion 110(R)(2), right portion 110(R)(3), and bottom portion 110(R)(4) of right panel mask 110(R). Thus, for a two-panel HMD 100 rendering a left panel mask 110 (L) and a right panel mask 110 for a given frame 106, a total of eight target size values ​​may be determined in block 402. The determination of the target size value(s) in block 402 may depend on whether reprojection is currently being used to render the series of frames 106. If reprojection is ongoing (e.g., if the given frame 106 is part of a series of frames 106 being rendered using reprojection), the target size value(s) may be determined based at least in part on rotation data provided by a head tracking module of the HMD 100. If reprojection is not ongoing, the target size value(s) may be set to a value(s) of zero in block 402. As mentioned, the target size values ​​of individual portions of panel mask 110 may be fixed between a minimum size value (e.g., "0") and a maximum size value (e.g., "1"), and thereby may be set to any value between these two limits (e.g., "0" and "1"), inclusive. In particular, the target size value may be calculated as a value corresponding to the amount by which HMD 100 has rotated while reprojection is in progress. For example, a greater amount of rotation of HMD 100 corresponds to a target size value closer to the maximum size value (e.g., "1"), and a smaller amount of rotation of HMD 100 corresponds to a target size value closer to the minimum size value (e.g., "0").

[0032] At 404, logic of HMD 100 may adjust a size value associated with at least a portion of panel mask 110 from the existing size value to the target size value determined at block 402, assuming the target size value is not equal to the existing size value. That is, if the target size value is equal to the existing size value, the logic may not perform a size adjustment at block 404. Thus, if reprojection is in progress and the logic determines that the size value associated with the portion of the panel mask is set to an existing size value that is less than the target size value, the adjustment at block 404 may increase the size of at least a portion of panel mask 110 from its current size to an increased size. If reprojection is not in progress (target size value is zero at block 402) and the existing size value is currently non-zero, or if reprojection is in progress and the non-zero target size value is less than the existing size value, the adjustment at block 404 may decrease the size of at least a portion of panel mask 110 from its current size to a decreased size. The downward size adjustment to the reduced size in block 404 may be a slow interpolation of small amounts (e.g., erp values ​​of 0.005 per frame) from the current size value to a smaller target value (e.g., a minimum value or other target value). In some embodiments, the downward size adjustment in block 404 is further conditioned on the size of the portion of panel mask 110 not increasing for a predetermined number of consecutive frames 106 preceding the current frame 106. For example, if the size of the portion of panel mask 110 has not increased for the last 10 consecutive frames, its size may be adjusted downward to the reduced size. Alternatively, if this consecutive frame criterion is not met, the size value may not be adjusted in block 404.

[0033] At 406, logic in the HMD 100 can render a frame 106 of the series of frames 106 onto the display panel 102 with a panel mask 110 rendered around the periphery of the display panel 102, where at least a portion of the panel mask 110 is rendered at a size (e.g., an adjusted size) according to the target size value.

[0034] Process 400 enables panel mask 110 to expand to cover areas of display panel 102 where the undesired visual artifacts would otherwise appear, thereby hiding undesirable visual artifacts that may occur during reprojection in conjunction with head rotation. This is shown, by way of example and not limitation, in Figures 1A and 1B, where a portion of panel mask 110 on each display panel 102 is increased in size from its current size to an increased size and rendered along with frame 106 on display panel(s) 102.

[0035] 5 is a flow diagram of a more detailed exemplary process 500 for dynamically resizing a portion of a panel mask 110 on a display panel 102 of an HMD 100, according to an embodiment disclosed herein. For discussion purposes, the process 500 will be described with reference to the previous figure.

[0036] At 502, a frame 106 of the series of frames 106 is processed for rendering on the display panel(s) 102 of the HMD 100. At this point, the panel mask(s) 110 that are rendered on the display panel(s) 102 along with the frame 106 may have size values ​​associated with those portions of the panel mask(s) 110 set to existing size values ​​that define whether those portions are in a fully contracted state, a fully expanded state, or somewhere in between. If the panel mask(s) 110 have not previously expanded to an increased size, the existing size value of each portion may be set to the minimum size value (e.g., a size value of zero) that corresponds to the fully contracted state.

[0037] At 504, logic in the HMD 100 may determine whether a most recently rendered frame 106 in the series of frames 106 is a reprojected frame 106(R) derived from pixel data received from an application associated with a previously rendered actual frame 106(A). In other words, logic in the HMD 100 determines whether reprojection is currently occurring. If at 504 it is determined that the series of frames 106 are being rendered on the display panel(s) 102 of the HMD 100 using reprojection, process 500 may follow the “YES” route from block 504 to block 506.

[0038] In block 506, logic in the HMD 100 can determine target size values ​​for individual portions of the panel mask 110 based at least in part on rotation data provided by the head tracking module of the HMD 100. For example, four target size values ​​fixed between 0.0 and 1.0 per direction can be determined based on the rotation of the HMD 100. That is, a first target size value between 0 and 1 (inclusive) can be determined for the left portion of the panel mask 110, a second target size value between 0 and 1 (inclusive) can be determined for the top portion of the panel mask 110, a third target size value between 0 and 1 (inclusive) can be determined for the right portion of the panel mask 110, and a fourth target size value between 0 and 1 (inclusive) can be determined for the bottom portion of the panel mask 110. This can be done for each panel mask 110, meaning a total of eight target size values ​​can be determined in block 506 for the left and right panel masks 110(L) and 110(R). Generally, the direction and amount of rotation of the HMD 100 defines the portion of the panel mask 110 and the target size value of that portion of the panel mask 110, respectively.

[0039] As shown in sub-block 508, logic in the HMD 100 may determine, based on rotation data provided by the head tracking module of the HMD 100, the amount of rotation of the HMD 100 in each direction (e.g., left, right, up, and down) over a period of time since a rendered frame 106 in the series of frames 106 was last rendered. Any suitable method for determining the amount of rotation in any unit of measure may be used for the determination in block 508. For example, the number of degrees of rotation between frames may be determined as the amount of rotation. In some embodiments, the amount of rotation determined in block 508 may be in the form of an angular velocity. The directionality of the rotation of the HMD 100 may be determined with reference to positive or negative horizontal and vertical directions or other suitable forms of direction measurement, and the directionality of the rotation of the HMD 100 may be resolved into component directions (e.g., left, right, up, and / or down in the XY plane). Thus, the logic of block 506 may include determining a portion(s) of the multiple portions of panel mask(s) 110 that corresponds to the direction in which the HMD 100 has rotated. Generally, the greater the amount of rotation of the HMD 100, the greater the target size value, and therefore the greater the amount of rotation, the more of the image area 108 on the display panel 102 is covered by the panel mask 110. In this sense, there may be a threshold amount of rotation that corresponds to a maximum size value for the portion of the panel mask 110, and if the amount of rotation of the HMD 100 is equal to or greater than this threshold amount, the maximum size value may be selected as the target size value in block 506. If the amount of rotation of the HMD 100 is less than this threshold amount, the target size value is determined to be less than the maximum size value. In some embodiments, the target size value may be determined based on the relative difference between the reprojection transformations used for the previous frame 106 and the current frame 106.That is, instead of, or in addition to, determining the amount of rotation of the HMD 100, the relative difference between the reprojection transformations used during the previous frame 106 and the current frame 106 can be used to drive the calculation of the target size value based on the idea that the greater the difference between the reprojection transformations, the more portions of the display panel will exhibit undesirable visual artifacts (e.g., strobe flashes between black pixels and illuminated / valid pixels).

[0040] If it is determined at 504 that the series of frames 106 are not being rendered on the display panel(s) 102 of the HMD 100 using reprojection, the process 500 can take the “NO” route from block 504 to block 510. After the previous iteration, this determination at block 504 can be a determination that reprojection has stopped. For example, after rendering frame 106 and proceeding to process the next frame at 502, the logic of the HMD 100 can determine that the series of frames 106 are being rendered without reprojection (e.g., without the reprojected frame 106(R)), indicating that the use of reprojection has stopped. In other words, one or more future frames rendered on the display panel 102 may not include any reprojected frame 102(R), indicating that the use of reprojection has stopped or that reprojection is not currently occurring for another reason. This can occur if the application reaches its frame rate after previously not reaching its frame rate.

[0041] In 510, logic in the HMD 100 can set the target size values ​​to zero (or a minimum size value) for individual portions of the panel mask 110. For example, four target size values ​​corresponding to the left, top, right, and bottom portions of the panel mask 110 can be set to zero in block 510.

[0042] At 512, following either blocks 506 / 508 or block 510, the logic of the HMD 100 may determine, for each individual portion of the panel mask 110, whether the target size value is greater than the existing size value of the portion of the panel mask 110. If at block 512 it is determined that the target size value is greater than the existing size value, the process 500 may follow the "YES" route from block 512 to block 514.

[0043] At 514, logic in the HMD 100 may adjust (or change) a size value associated with the portion of the panel mask 110 from an existing size value to a target size value, increasing the size of the portion of the panel mask 110 from its current size to an increased size. As shown, this adjustment to increase size may be based on a target size value that is greater than the existing size value, indicating that the portion of the panel mask 110 is pulled inward to cover more of the image region 108. In some embodiments, the size adjustment at block 514 is temporarily smoothed to mitigate abrupt field of view changes.

[0044] At 516, logic in the HMD 100 may render the frame 106 on the display panel(s) 102 with panel mask(s) 110 rendered around the display panel(s) 102, where at least a portion of the panel mask is rendered at a size according to the target size value, and at block 518, the next frame 106 in the series of frames 106 may be processed by repeating from block 502 for the next frame. Following block 514, the panel mask 110 may be rendered at an increased size at block 516 by increasing from the existing size value to the target size value. It should also be understood that in the case of an HMD 100 with a pair of display panels (e.g., a left display panel 102(L) and a right display panel 102(R)), the algorithm of FIG. 5 may be performed independently for each panel mask 110(L) and 110(R). In practice, the same portion of each mask may be adjusted in the same or similar manner in response to a rotation of a particular HMD 100 during reprojection, such as when a leading edge portion of the panel mask 110 on each display panel 102 expands inward toward the center of each display panel 102, as illustrated in the examples of FIGS. 1A and 1B. Furthermore, the frame 106 for which the panel mask 110 is rendered according to the newly determined target size value may be the actual frame 106(A) or the reprojected frame 106(R). In at least one example, the determination in block 504 that a reprojection is occurring is based on the previously rendered reprojected frame 106(R). Thus, the subsequent frame may be the actual frame 106(A) received from the application, in which case the panel mask 110 is rendered in block 516 along with the actual frame 106(A) according to the calculated target size value. In other words, when a reprojection is first detected, the initial frame 106 rendered with the resized panel mask 110 may be the next frame, which may be the actual frame 106(A) received from the application.However, in some embodiments, for example, if multiple reprojected frames 106(R) are rendered between successive actual frames 106(A) (e.g., if an application is outputting frames at 30 frames per second, two reprojected frames 106(R) are rendered for every actual frame 106(A)), the first frame 106 rendered with the increased-size panel mask 110 may be the next reprojected frame 106(R).

[0045] Referring again to block 512, if it is determined at block 512 that the target size value is not greater than the existing size value, process 500 may follow the “NO” route from block 512 to block 520. At 520, logic in HMD 100 may determine whether the target size value (of the respective portion of panel mask 110) is less than the existing size value. If the answer at block 520 is “no,” process 500 may follow the “NO” route from block 520 to block 522 based on a determination that the target size value is equal to the existing size value. At 522, no adjustment is made to the size value of the given portion of panel mask 110, and the frame is rendered at block 516 without any adjustment to the size of the portion of panel mask 110. In other words, if it is determined that panel mask 110 already covers the portion that it should cover, no adjustment is made. On the other hand, if block 520 determines that the target size value of the portion of panel mask 110 is less than the existing size value, process 500 may follow the “YES” route from block 520 to block 524 .

[0046] At 524, logic in the HMD 100 may determine whether a predetermined number of frames 106 (e.g., N frames, where N is equal to 5, 10, etc.) have been rendered consecutively without increasing the size of that portion of the panel mask 110. If the size of that portion of the panel mask 110 has increased over the last N consecutive frames 106, the process 500 may take the “NO” route from block 524 to block 522, in which case no adjustments are made to that portion of the panel mask 110 and to block 516, where frames are rendered without adjusting the size of that portion of the panel mask 110. This may be based on the idea that it may be beneficial to have some confidence that reprojection has stopped before shrinking the panel mask 110 to make previously covered portions of the image area 108 visible. This confidence may be based on a certain number of consecutive frames 106(A) being rendered without increasing the size of that portion of the panel mask 110. Thus, if there has not been any increase in the size of the portion of the panel mask 110 over the last N consecutive frames 106, the process 500 may follow the "YES" route from block 524 to block 526.

[0047] At 526, the size value may be decremented (e.g., gradually adjusted downward) toward a smaller target value (e.g., a target size value, or otherwise, a minimum size value of “0” corresponding to a fully contracted state of the panel mask 110). This may allow the size of the portion of the panel mask 110 to be decremented from the existing size value to a target size value that is less than the existing size value over a period of time (e.g., approximately 2 seconds), thereby decreasing the size of the portion of the panel mask 110 from its current size to a smaller size over that period of time. Thus, the decrement at block 526 over multiple frames will ultimately decrease the size of the previously enlarged portion of the panel mask 110. As the head rotation slows and / or reprojection stops, the previously enlarged portion of the panel mask 110 may eventually decrease to a minimum size over multiple iterations of the process 500. This allows the rate of contraction of the panel mask 110 to be controlled at a rate imperceptible to the human eye. In this manner, the contraction of the panel mask 110 should not be distracting to the user 104. In some embodiments, the decrement operation at block 526 is a slow interpolation step that decrements the existing size value by a relatively small amount (e.g., an erp value of 0.005 per frame) to a smaller target value. In other words, the logic may be configured to avoid shrinking the portion of panel mask 110 faster than a maximum shrink rate; that is, if the user's 104 head rotation suddenly slows significantly, instead of reducing the size of the portion of panel mask 110 to the determined target size value, the existing size value may be decremented to an intermediate size value that is greater than the target size value and less than the existing size value. In some embodiments, if the target size value is between the smaller size value and the existing size value, the decrement at block 526 decrements to the target size value.After decrementing the existing size value (e.g., reducing the existing size value by a particular percentage (e.g., 5%)), process 500 proceeds from block 526 to block 516, where frame 106 is rendered on display panel 102 along with the panel mask, where the portion of panel mask 110 whose size has been decremented is rendered at a reduced size compared to the previous size of the portion of panel mask 110.

[0048] During subsequent iterations of process 500, block 504 may determine whether reprojection is still being used to render the series of frames 106 or whether reprojection has stopped, which may be understood to dictate whether individual portions of panel mask 110 are dynamically increased, decreased, or left unchanged compared to the previous size adjustment. Generally, appropriate portions of panel mask 110 increase in size during reprojection in conjunction with further HMD 100 rotation, and as HMD 100 rotation slows and / or reprojection stops, the portions of panel mask 110 decrease in size, allowing more image area 108 to be displayed on display panel 102.

[0049] FIG. 6 illustrates that exemplary components of an HMD 600, such as a VR headset, may be incorporated according to embodiments disclosed herein. The HMD 600 may be the same as or similar to the HMD 100 referenced in the previous figure, and thus the components of the HMD 600 shown in FIG. 6 may be implemented in the HMD 100. The HMD 600 may be implemented as a standalone device worn by the user 104 (e.g., on the head of the user 104). In some embodiments, the HMD 600 may be head-mountable, such as by allowing the user 104 to secure the HMD 600 to their head using a fastening mechanism (e.g., an adjustable band) sized to fit around the user's 104's head. In some embodiments, the HMD 600 comprises a virtual reality (VR) or augmented reality (AR) headset including near-eye or near-to-eye display(s). Accordingly, the terms "wearable device," "wearable electronic device," "VR headset," "AR headset," and "head-mounted display (HMD)" may be used interchangeably herein to refer to device 600 of Figure 6. However, it should be understood that these types of devices are merely examples of HMD 600, and that HMD 600 may be implemented in a variety of other form factors.

[0050] In the illustrated implementation, the HMD 600 includes one or more processors 602 and memory 604 (e.g., computer-readable medium 604). In some implementations, the processor(s) 602 may include a central processing unit (CPU), a graphics processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing devices or components known in the art. Alternatively or additionally, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) 602 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0051] Memory 604 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Memory 604 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium that can be accessed by processor(s) 602 to execute instructions stored in memory 602. In one basic implementation, the CRSM may include random access memory (“RAM”) and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and that can be accessed by the processor(s) 602.

[0052] In general, the HMD 600 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to perform the techniques, functions, and / or operations described herein. The computer-readable medium 604 is shown as including various modules, such as instructions, data storage, etc., which may be configured to execute on the processor(s) 602 to perform the techniques, functions, and / or operations described herein. While several exemplary functional modules are shown as stored on the computer-readable medium 604 and executable on the processor(s) 602, the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-chip (SOC) and / or other logic.

[0053] The operating system module 606 may be configured to manage hardware within the HMD 600 and coupled to the HMD 600 for the convenience of other modules. Additionally, in some cases, the HMD 600 may include one or more applications 608 stored in the memory 604 or otherwise accessible to the HMD 600. In this implementation, the application(s) 608 include a game application 610. However, the HMD 600 may include any number or type of applications and is not limited to the specific example shown. The game application 610 may be configured to initiate gameplay of a video-based interactive game (e.g., a VR game) playable by the user 104 and output a frame (e.g., actual frame 106(A)) to be rendered on the display panel of the HMD 100. The compositor 611 may be configured to render and dynamically adjust the panel mask 110 with each frame as a later rendering operation. Thus, the compositor 611, together with associated logic of the HMD 100, can perform the dynamic panel mask adjustment techniques disclosed herein.

[0054] Generally, the HMD 600 has input devices 612 and output devices 614. The input devices 612 may include control buttons. In some implementations, one or more microphones may function as the input devices 612 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) may function as the input devices 612 to receive gestural input, such as hand and / or head movements of the user 104. In some embodiments, additional input devices 612 may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, etc. In other embodiments, the HMD 600 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 600 may implement a relatively simple form of input device 612, a network interface (wireless or wired-based), power, and processing / memory capabilities. For example, a limited set of one or more input components (e.g., dedicated buttons for initiating configuration, powering on / off, etc.) may be employed to subsequently enable use of the HMD 600. In one implementation, the input device(s) 612 may include controls such as basic volume control button(s) for increasing / decreasing the volume, as well as a power button and a reset button.

[0055] The output device(s) 614 may include display(s) 616, which may be the same as or similar to the display panel(s) 102 described with reference to the previous figure. The output device(s) 614 may further include, but are not limited to, light elements (e.g., LEDs), vibrators for producing tactile sensations, speaker(s) (e.g., headphones), etc. For example, there may also be simple light elements (e.g., LEDs) to indicate a status such as when the power is on.

[0056] The HMD 600 may further include a wireless unit 618 coupled to an antenna 620 to facilitate wireless connection to a network. The wireless unit 618 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the HMD 600 may further include a physical port to facilitate a wired connection to a network, connected peripherals (including PCs, game consoles, etc.), or plug-in network devices that communicate with other wireless networks.

[0057] The HMD 600 may further include an optical subsystem 622 that directs light from the electronic display(s) 616 to the user's eye(s) using one or more optical elements. The optical subsystem 622 may include various types and combinations of different optical elements, including, but not limited to, apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 622 may have one or more coatings, such as an anti-reflective coating. The magnification of image light by the optical subsystem 622 may allow the electronic display(s) 616 to be physically smaller, lighter, and consume less power than larger displays. Furthermore, the magnification of image light may increase the field of view (FOV) of the displayed content (e.g., an image). For example, the FOV of the displayed content may be such that the displayed content is presented using nearly all (e.g., 120-150 degrees diagonal), or in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., an FOV of about 40 degrees). The optical subsystem 622 may be designed to correct one or more optical errors, such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, coma, field curvature, and astigmatism. In some embodiments, content provided to the electronic display(s) 616 for display is pre-distorted, and the optical subsystem 622 corrects the distortion when receiving image light from the electronic display(s) 616 generated based on the content.

[0058] The HMD 600 may further include one or more sensors 624, such as sensors used to generate movement, position, and orientation data. These sensors 624 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other movement, position, and orientation sensors. The sensors 624 may also include a sub-portion of sensors, such as a set of active or passive markers that can be viewed externally by a camera or color sensor to generate movement, position, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared or visible light), that, when viewed by an external camera or illuminated with light (e.g., infrared or visible light), may provide one or more reference points for interpretation by software to generate movement, position, and orientation data. The HMD 600 may include a light sensor that is sensitive to light (e.g., infrared or visible light) projected or transmitted by a base station within the HMD 600's environment.

[0059] In one example, the sensor(s) 624 may include an inertial measurement unit (IMU) 626. The IMU 626 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU 626, or some combination thereof. Based on the measurement signals, such motion-based sensors, such as the IMU 626, can generate calibration data that indicates an estimated position of the HMD 600 relative to an initial position of the HMD 600. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, roll). The IMU 626 may, for example, rapidly sample the measurement signals and calculate an estimated position of the HMD 600 from the sampled data. For example, the IMU 626 may integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 600. A reference point is a point that can be used to describe the position of the wearable device 702. A reference point may generally be defined as a point in space, but in various embodiments, the reference point is defined as a point within the HMD 600 (e.g., the center of the IMU 626). Alternatively, the IMU 626 provides sampled measurement signals to an external console (or other computing device), which determines the calibration data.

[0060] The sensor 624 can operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data can be generated at a rate of 1000 Hz (or one sensor reading every millisecond). In this manner, 1000 readings are taken per second. If the sensor generates this much data at this rate (or even faster), the data set used to predict movement can become very large, even over a relatively short period of time, such as tens of milliseconds.

[0061] As mentioned, in some embodiments, the sensor 624 may include a light sensor that is sensitive to light emitted by a base station in the environment of the HMD 600 for purposes of tracking the position and / or orientation, posture, etc. of the HMD 600 in 3D space. The calculation of the position and / or orientation may be based on timing characteristics of the light pulses and the presence or absence of light detected by the sensor 624.

[0062] The HMD 600 may further include an eye tracking module 628. A camera or other optical sensor within the HMD 600 can capture image information of the user's eyes, and the eye tracking module 628 can use the captured information to determine the interpupillary distance, interocular distance, and the three-dimensional (3D) position of each eye relative to the HMD 600 (e.g., for distortion adjustment purposes), including the magnitude of twist and rotation (i.e., roll, pitch, yaw) and gaze direction of each eye. In one example, infrared light is emitted within the HMD 600 and reflected from each eye. The reflected light is received or detected by a camera of the HMD 600 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 104 can be used by the eye tracking module 628. Thus, the eye tracking module 628 can track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) for each eye and can combine at least a subset of the tracked quantities from the two eyes of the user 104 to estimate a point of gaze (i.e., a 3D point or location in the virtual scene at which the user is viewing). For example, the eye tracking module 628 can integrate information from past measurements, measurements identifying the position of the user's 104's head, and 3D information describing the scene presented by the electronic display(s) 616. Thus, the position and orientation information of the user's 104's eyes is used to determine a point of gaze in the virtual scene presented by the HMD 600 at which the user 104 is viewing.

[0063] The HMD 600 may further include a head tracking module 630. The head tracking module 630 can utilize one or more of the sensors 624, as described above, to track head movement, including head rotation, of the user 104. For example, the head tracking module 630 can track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of the HMD 600. These calculations can be performed for each frame 106 of a series of frames 106, allowing the application 608 to determine how to render a scene in the next frame 106 (even for a reprojected frame 106(R)) depending on the head position and orientation. In some embodiments, the head tracking module 630 is configured to predict the future position and / or orientation of the HMD 600 based on current and / or past data. This is because the application is required to render a frame 106 before the user 104 actually sees the light (and therefore the image) on the display(s) 616. Thus, the next frame 106 may be rendered based on this future prediction of head position and / or orientation made at an earlier point in time, such as approximately 25-30 milliseconds (ms) before rendering frame 106. The rotation data provided by the head tracking module 630 may be used to determine both the direction of rotation of the HMD 600 and the amount of rotation of the HMD 600 in any suitable units of measure. For example, the direction of rotation may be simplified and output in terms of positive or negative horizontal directions and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation may be expressed in units of degrees, radians, etc. Angular velocity may be calculated to determine the rate of rotation of the HMD 600.

[0064] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the particular features described. Rather, the particular features are disclosed as example forms of implementing the claims. The inventions described in the original claims of this application are set forth below. [1] A head-mounted display (HMD), a pair of display panels including a left display panel and a right display panel; one or more processors; a head tracking module that provides rotational data indicative of rotational movement of the HMD to another processor; It is logic, determining that a frame of a series of frames rendered on the pair of display panels is a reprojected frame derived from pixel data received from an application and associated with a previously rendered actual frame, the frame being rendered with a left panel mask around the periphery of the left display panel and a right panel mask around the periphery of the right display panel; determining an amount of rotation the HMD has rotated over a period of time since rendering the frame based on the rotation data provided by the head tracking module; determining a direction in which the HMD has rotated based on the rotation data; determining a portion of the left panel mask and a portion of the right panel mask that each correspond to the direction to which the HMD has rotated; determining a first target size value for the portion of the left panel mask and a second target size value for the portion of the right panel mask based on the amount of rotation of the HMD; determining that the first target size value is not equal to a first existing size value for the portion of the left panel mask; determining that the second target size value is not equal to a second existing size value for the portion of the right panel mask; and logic configured to: render a next frame of the series of frames on the pair of display panels with the left panel mask around the periphery of the left display panel and the right panel mask around the periphery of the right display panel, wherein the portion of the left panel mask is rendered at a first adjusted size corresponding to the first target size value and the portion of the right panel mask is rendered at a second adjusted size corresponding to the second target size value. [2] the next frame is an actual frame received from the application, and the logic determining, after rendering the next frame, that one or more future frames of the series of frames to be rendered on the pair of display panels does not include any reprojected frames; The HMD described in [1] is further configured to decrement size values ​​for each of the portion of the left panel mask and the portion of the right panel mask from the first and second target size values ​​to a minimum size value over a period of time, thereby reducing the size of the portion of the left panel mask and the portion of the right panel mask to a minimum size over the period of time. [3] An HMD as described in [1], wherein the portion of the left panel mask corresponds to the portion rendered at the front edge of the left display panel in the direction in which the HMD has rotated, and the portion of the right panel mask corresponds to the portion rendered at the front edge of the right display panel in the direction in which the HMD has rotated. [4] 1. A method comprising: determining, by one or more processors, that a series of frames are being rendered on a display panel of a head-mounted display (HMD) using reprojection, where the reprojection generates reprojected frames from pixel data associated with actual frames received from an application that have already been rendered on the display panel; determining a target size value for at least a portion of a panel mask based at least in part on rotational data provided by a head tracking module of the HMD; determining that a size value associated with at least the portion of the panel mask is set to an existing size value that is less than the target size value; adjusting the size value from the existing size value to the target size value to increase a size of at least the portion of the panel mask from a current size to an increased size; rendering a frame of the series of frames on the display panel with the panel mask rendered around the periphery of the display panel, at least the portion of the panel mask being rendered at the increased size according to the target size value. [5] After rendering the frames, determining that the series of frames has been rendered without the reprojected frames, which indicates that the use of reprojection has stopped; and [4] The method of [4], further comprising: decrementing the size value from the target size value to a smaller size value that is less than the target size value over a period of time, and decreasing the size of at least the portion of the panel mask from the increased size to the decreased size over the period of time. [6] After rendering the frame, increasing a size of at least the portion of the panel mask to determine that a predetermined number of frames have been rendered consecutively; [5] The method of [5], wherein the decrementing of the size value over the period of time is based at least in part on determining that the predetermined number of frames have been rendered consecutively without increasing the size of at least the portion of the panel mask. [7] determining the target size value for at least the portion of the panel mask based at least in part on the rotation data provided by the head tracking module of the HMD, determining an amount of rotation of the HMD since a rendered frame of the series of frames was last rendered based on the rotation data; and determining the target size value for at least the portion of the panel mask based on the amount of rotation. [8] determining that the amount of rotation of the HMD since the rendered frame was last rendered is greater than or equal to a threshold amount; [7] The method of [7], wherein determining the target size value for at least the portion of the panel mask includes selecting a maximum size value as the target size value for at least the portion of the panel mask. [9] determining a direction in which the HMD has rotated since a rendered frame of the sequence of frames was last rendered based on the rotation data; determining that at least the portion of the panel mask corresponds to the orientation to which the HMD has rotated; The method of [4], wherein adjusting the size value associated with at least the portion of the panel mask is based at least in part on determining that at least the portion of the panel mask corresponds to the direction in which the HMD has rotated.

[10] [9] The method of [9], wherein at least the portion of the panel mask corresponds to a portion of the panel mask being rendered at the front edge of the display panel in the direction to which the HMD is rotated.

[11] [4] The method of claim 4, wherein the size value associated with at least the portion of the panel mask is adjustable between a minimum size value and a maximum size value, the minimum size value corresponding to a fully contracted state of at least the portion of the panel mask, and the maximum size value corresponding to a fully expanded state of at least the portion of the panel mask.

[12] [4] The method according to [4], wherein the frame rendered on the display panel is the first actual frame rendered after rendering of a reprojected frame among the reprojected frames.

[13] The display panel is a first display panel of a pair of display panels, the pair of display panels including the first display panel and a second display panel, and the method includes, for the second display panel of the pair of display panels: determining a second target size value for at least a portion of a second panel mask based at least in part on the rotation data; determining that a second size value associated with at least the portion of the second panel mask is set to a second existing size value that is less than the second target size value; adjusting the second size value from the second existing size value to the second target size value to increase a size of at least the portion of the second panel mask from a second current size to a second increased size; [4] The method of [4], further comprising: rendering the frame of the series of frames on the second display panel with the second panel mask rendered around the periphery of the second display panel, wherein at least the portion of the second panel mask is rendered at the second increased size according to the second target size value.

[14] A head-mounted display (HMD), A display panel; one or more processors; It is logic, determining that a series of frames are being rendered on the display panel using reprojection, the reprojection generating reprojected frames from pixel data associated with actual frames received from an application that have already been rendered on the display panel; determining a target size value for at least a portion of a panel mask based at least in part on whether the series of frames is rendered using reprojection; determining that a size value associated with at least the portion of the panel mask is set to an existing size value that is not equal to the target size value; adjusting the size value from the existing size value to the target size value to increase or decrease a size of at least the portion of the panel mask from a current size to an adjusted size; and logic configured to: render a frame of the series of frames on the display panel with the panel mask rendered around the periphery of the display panel, at least a portion of the panel mask being rendered at the adjusted size according to the target size value.

[15] Determining whether the series of frames is rendered on the display panel using the reprojection includes determining that the series of frames is rendered on the display panel using the reprojection, and the logic further comprising: After rendering the frames, determining that the series of frames has been rendered without the reprojected frames, which indicates that the use of reprojection has stopped; and The HMD described in

[14] is further configured to decrement the size value from the target size value to a smaller size value less than the target size value over a period of time, and to decrease the size of at least the portion of the panel mask from the adjusted size to a reduced size over the period of time.

[16] a head tracking module for providing rotational data indicative of rotational movement of the HMD to another processor, wherein determining the target size value comprises: determining an amount of rotation of the HMD since a rendered frame of the series of frames was last rendered based on the rotation data;

[14] The HMD described in

[14] , further comprising: determining the target size value for at least the portion of the panel mask based on the amount of rotation.

[17] The logic is: determining a direction in which the HMD has rotated since the rendered frame was last rendered based on the rotation data; further configured to determine that at least the portion of the panel mask corresponds to the orientation to which the HMD has been rotated;

[16] The HMD described in

[16] , wherein adjusting the size value associated with at least the portion of the panel mask is based at least in part on determining that at least the portion of the panel mask corresponds to the direction in which the HMD has rotated.

[18] determining that the size value is set to the existing size value that is not equal to the target size value includes determining that the target size value is greater than the existing size value;

[14] The HMD described in

[14] , wherein adjusting the size value from the existing size value to the target size value increases the size of at least the portion of the panel mask from the current size to an increased size.

[19]

[14] The HMD described in

[14] , wherein the frame rendered on the display panel is the first actual frame rendered after rendering of a reprojected frame among the reprojected frames.

[20] determining whether the series of frames has been rendered on the display panel using the reprojection includes determining that the series of frames has not been rendered on the display panel using the reprojection; The HMD of

[14] , wherein determining the target size value for at least the portion of the panel mask includes selecting a minimum size value as the target size value for at least the portion of the panel mask.

Claims

1. a head-mounted display (HMD) having a display panel; determining that the sequence of frames is being rendered using reprojection, the reprojection being based at least in part on rotation data provided by a head tracking module of the HMD; increasing a size of at least a portion of a panel mask from a current size to an increasing size inward toward a center of the display panel based at least in part on determining that the series of frames is being rendered using the reprojection; logic configured to render the panel mask on a periphery of the display panel and render a frame of the series of frames on the display panel excluding the periphery, wherein at least the portion of the panel mask is rendered at the increased size; Equipped with The logic is: after rendering the frame, not decreasing the size of at least the portion of the panel mask from the increased size to a decreased size until determining that the use of the reprojection has stopped or that the rotation of the HMD has slowed; system.

2. The logic is: determining that the use of reprojection has ceased after rendering the frame; reducing a size of at least the portion of the panel mask from the increased size to a decreased size; rendering subsequent frames of the series of frames on the display panel with the panel mask rendered on the periphery of the display panel, at least the portion of the panel mask being rendered at the reduced size; and The system of claim 1 , further configured to:

3. The logic is: after rendering the frame, determining that a predetermined number of frames have been rendered consecutively without increasing the size of at least the portion of the panel mask; and further configured to: reducing the size of at least the portion of the panel mask is based at least in part on determining that the predetermined number of frames have been rendered consecutively without increasing the size of at least the portion of the panel mask. The system of claim 2 .

4. The logic is: determining a first amount of rotation of the HMD since a rendered frame of the sequence of frames was rendered based at least in part on the rotation data; determining a second amount to increase the size of at least the portion of the panel mask based at least in part on the first amount of rotation; and further configured to: increasing the size of at least the portion of the panel mask includes increasing the size of at least the portion of the panel mask by the second amount. The system of claim 1 .

5. The logic is: determining a direction in which the HMD has rotated since a rendered frame of the sequence of frames was rendered based at least in part on the rotation data; and determining that at least the portion of the panel mask corresponds to an orientation to which the HMD has been rotated; The system of claim 1 further configured to:

6. The system of claim 5 , wherein at least the portion of the panel mask corresponds to a portion of the panel mask that is rendered at a front edge of the display panel in the orientation to which the HMD has been rotated.

7. 2. The system of claim 1, wherein the size of at least the portion of the panel mask is adjustable between a minimum and a maximum, the minimum corresponding to a fully contracted state of at least the portion of the panel mask and the maximum corresponding to a fully expanded state of at least the portion of the panel mask.

8. the display panel is a first display panel of a pair of display panels, the pair of display panels including the first display panel and a second display panel; The logic is: increasing a size of at least a portion of a second panel mask from a second current size to a second increased size based at least in part on determining that the series of frames is being rendered using the reprojection; rendering the frames of the series of frames on the pair of display panels with the second panel mask rendered on a periphery of the second display panel, at least the portion of the second panel mask being rendered at the second increased size; The system of claim 1 further configured to:

9. determining, by one or more processors, that the series of frames are being rendered using reprojection, the reprojection being based at least in part on rotation data provided by a head-mounted display (HMD) having a display panel; increasing a size of at least a portion of a panel mask from a current size to an increased size inward toward a center of the display panel based at least in part on determining that the series of frames is being rendered using the reprojection; Rendering the panel mask on a periphery of the display panel and rendering a frame of the series of frames on the display panel excluding the periphery, wherein at least the portion of the panel mask is rendered at the increased size; after rendering the frame, not decreasing the size of at least the portion of the panel mask from the increased size to a decreased size until determining that the use of the reprojection has stopped or that the rotation of the HMD has slowed; A method comprising:

10. determining that the reprojection has been disabled after rendering the frame; decreasing the size of at least the portion of the panel mask from the increased size to a decreased size; rendering subsequent frames of the series of frames on the display panel with the panel mask rendered on a periphery of the display panel, at least the portion of the panel mask being rendered at the reduced size; The method of claim 9 further comprising:

11. determining a first amount of rotation of the HMD since a rendered frame of the sequence of frames was rendered based at least in part on the rotation data; determining a second amount to increase the size of at least the portion of the panel mask based at least in part on the first amount of rotation; Further comprising: Increasing the size of at least the portion of the panel mask comprises increasing the size of at least the portion of the panel mask by the second amount.

10. The method of claim 9.

12. the display panel is a first display panel of a pair of display panels, the panel mask is a first panel mask, and the pair of display panels includes the first display panel and a second display panel; The method increasing a size of at least a portion of a second panel mask from a second current size to a second increased size based at least in part on determining that the series of frames is being rendered using the reprojection; rendering a frame of the series of frames on the pair of display panels with the second panel mask rendered on a periphery of the second display panel; Further comprising: at least the portion of the second panel mask is rendered at the second increased size; 10. The method of claim 9.

13. determining a direction in which the HMD has rotated since a rendered frame of the sequence of frames was rendered based at least in part on the rotation data; and determining that at least the portion of the first panel mask corresponds to a portion of the first panel mask that is rendered at a front edge of the first display panel in a rotated orientation of the HMD; determining that at least a portion of the second panel mask corresponds to a portion of the second panel mask that is rendered at a front edge of a second display panel in a rotated orientation of the HMD; 13. The method of claim 12, further comprising:

14. a head-mounted display (HMD) having a display panel; one or more processors; a memory storing computer-executable instructions; A system comprising: The computer-executable instructions, when executed by the one or more processors, determining that the series of frames are being rendered using reprojection, the reprojection being based at least in part on rotation data provided by a head tracking module of the HMD; increasing a size of at least a portion of a panel mask from a current size to an increased size inward toward a center of the display panel based at least in part on determining that the series of frames is being rendered using the reprojection; and rendering the panel mask on a periphery of the display panel and rendering a frame of the series of frames on the display panel excluding the periphery, wherein at least a portion of the panel mask is rendered at the increased size; after rendering the frames, not decreasing the size from the increased size to the decreased size until determining that the series of frames has been rendered without using the reprojection or that the rotation of the HMD has slowed. A system that allows this.

15. The computer-executable instructions, when executed by the one or more processors, after rendering the frames, determining that the series of frames was rendered without using the reprojection; decreasing the size from the increased size to a decreased size; rendering subsequent frames of the series of frames on the display panel with the panel mask rendered on the periphery of the display panel, at least a portion of the panel mask being rendered at the reduced size; The system of claim 14 further comprising:

16. reducing the size includes decreasing a size of at least the portion of the panel mask from the increased size to the decreased size over a period of time. The system of claim 15.

17. The computer-executable instructions, when executed by the one or more processors, In the system, determining, prior to rendering the frame, a first amount of rotation of the HMD since a rendered frame of the series of frames was rendered based at least in part on the rotation data, wherein a second amount by which the size of at least the portion of the panel mask is increased is based at least in part on the first amount of rotation; After rendering the frame, determining that the series of frames is still being rendered using the reprojection; and determining a third amount of rotation of the HMD since the frame was rendered based at least in part on the rotation data, the third amount of rotation being greater than the first amount of rotation; increasing the size of at least the portion of the panel mask from the increased size to a second increased size greater than the increased size based at least in part on determining that the series of frames is still being rendered using the reprojection; rendering a second frame of the series of frames on the display panel with the panel mask rendered on the periphery of the display panel, wherein at least the portion of the panel mask is rendered at the second increased size; The system of claim 14 further comprising:

18. The computer-executable instructions, when executed by the one or more processors, In the system, prior to rendering the frame, determining a direction in which the HMD has rotated since a rendered frame of the sequence of frames was rendered based at least in part on the rotation data; determining that at least the portion of the panel mask corresponds to an orientation to which the HMD has been rotated; The system of claim 15 further comprising:

19. the size of at least the portion of the panel mask is adjustable between a minimum and a maximum, the minimum corresponding to a fully contracted state of at least the portion of the panel mask and the maximum corresponding to a fully expanded state of at least the portion of the panel mask; The system of claim 14 further comprising:

20. 20. The system of claim 18, wherein at least the portion of the panel mask corresponds to a portion of the panel mask being rendered at a front edge of the display panel in the orientation to which the HMD has rotated.

Images