Reprojection and jank at a head-mounted display device

Abstract

A head-mounted display device includes one or more position sensors and a processor. The processor can receive a rendered image for a current frame. The processor can receive position data from the one or more position sensors and determine an updated device pose based on the position data. The processor can apply a first spatial correction to color information in pixels of the rendered image at least in part by reprojecting the rendered image based on the updated device pose. The head-mounted display device can also include a display configured to apply a second spatial correction to the color information in the pixels of the reprojected rendered image at least in part by applying a wobble to the reprojected rendered image, thereby generating a sequence of wobbled pixel subframes for the current frame. The display can display the current frame by displaying the sequence of wobbled pixel subframes.

Description

BACKGROUND

[0001] Mixed reality and fully virtual reality head-mounted devices typically include a pair of near-eye displays located near each of the user's eyes. Fully virtual reality devices include non-transparent near-eye displays, while mixed reality devices include at least partially transparent near-eye displays. When a user uses a mixed reality device, images are projected via the near-eye displays to the user's eyes so that the images appear to be located in the user's physical three-dimensional environment, and images of the physical environment are visible through the displays when not occluded by the displayed images. For fully virtual reality displays, images appear to be projected into a fully virtual three-dimensional environment that visually replaces the user's physical environment. Mixed reality and fully virtual reality devices often take the form of eyeglasses or goggles that are worn by the user. For mixed reality devices, the device is able to display images so that the images appear to have world-locked positions when moving through the physical environment. SUMMARY

[0002] According to one aspect of the disclosure, a head-mounted display device is provided that includes one or more position sensors and a processor. The processor can be configured to receive a rendered image of a current frame that has been rendered at a remote computing device. The processor can also be configured to receive position data from the one or more position sensors. The processor can also be configured to determine an updated device pose of the head-mounted display device based on the position data. The processor can also be configured to apply a first spatial correction to color information in pixels of the rendered image at least in part by re-projecting the rendered image based on the updated device pose. The head-mounted display device can also include a display configured to apply a second spatial correction to color information in pixels of the rendered image at least in part by applying wobulation to the re-projected rendered image, thereby generating a sequence of wobulated pixel sub-frames for the current frame. The display can also be configured to display the current frame by displaying the sequence of wobulated pixel sub-frames.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any portion of this disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 A head-mounted display device according to one embodiment of the disclosure is shown.

[0005] Figure 2 It shows that according to Figure 1 A flowchart of an exemplary method that can be performed at one or more remote computing devices and a head-mounted display device, according to an embodiment of the present invention.

[0006] Figure 3 Schematic illustration based on Figure 2 The embodiments of the head-mounted display device and the remote computing device in one or more remote computing devices.

[0007] Figure 4A It shows that according to Figure 3 An exemplary reprojected rendered image generated based on the updated device posture of a head-mounted display device, as described in this embodiment.

[0008] Figure 4B It shows that it can be made by Figure 3 Examples of post-projection processing implemented in the embodiments to generate Figure 4A The reprojected rendered image.

[0009] Figure 5 It shows that according to Figure 3 The rendered image and the reprojected rendered image of the embodiment include color information and corrected color information, respectively.

[0010] Figure 6A-6D It shows that according to Figure 3 The embodiment is applied to the corresponding spatial offset of pixels in a sequence of four wobbly pixel subframes.

[0011] Figure 7A It shows that according to Figure 3 The embodiments are applied to the corresponding spatial offsets and sub-pixel colors of pixels in an exemplary sequence of twelve wobbly pixel subframes.

[0012] Figure 7B It shows that according to Figure 3 The embodiment is applied to the corresponding spatial offsets and sub-pixel colors of pixels in another exemplary sequence of twelve wobbly pixel subframes.

[0013] Figure 8A-8B It shows that according to Figure 1 The embodiments of the present invention include an exemplary architecture of a projector that replicates a pupil waveguide.

[0014] Figure 9A It shows that according to Figure 8A-8B A top view of an exemplary pupil replication waveguide of an embodiment.

[0015] Figure 9B It shows Figure 9A A cross-sectional view of an exemplary pupil replica waveguide.

[0016] Figure 9C A top view of another example pupil replication waveguide including a first redirecting grating and a second redirecting grating is shown in accordance with an embodiment of Figure 8A-8B

[0017] Figure 10A A top view of an example waveguide having a single entrance pupil is shown in accordance with an embodiment of Figure 1

[0018] Figure 10B A cross-sectional view of an example waveguide including multiple embedded gratings is shown in accordance with an embodiment of Figure 10A

[0019] Figure 11 Examples of a first pixel layout in which dithering is performed without field of view (FOV) stitching, a second pixel layout in which FOV stitching is performed without dithering, and a third pixel layout in which both dithering and FOV stitching are performed are shown in accordance with an embodiment of Figure 1

[0020] A schematic diagram of an example computing environment in which a head-mounted display device and a remote computing device of Figure 12 Figure 1 DETAILED DESCRIPTION

[0021] In existing mixed reality and fully virtual reality devices, the projector used to display images to a user can be relatively large. The size of such projectors can make it difficult to fit the projector within a comfortable eyeglass form factor for the user. Additionally, the projectors used in existing mixed reality devices can be heavier than desired. A heavy projector can make it uncomfortable for a user to wear a mixed reality device for extended periods of time.

[0022] The projectors included in existing mixed reality devices can also have a high power consumption. The high power consumption of existing projectors can reduce battery life and can require the user to charge the mixed reality device more frequently. Additionally, if the battery size is increased to account for the power requirements of the projector, the size and weight of the mixed reality device can be further increased.

[0023] To address the limitations of existing mixed reality devices, a head-mounted display device 10 in accordance with one example embodiment is shown in Figure 1 Figure 1 The head-mounted display device 10 of

[0024] ​​​​​​The head-mounted display device 10 can be configured in an augmented reality configuration to present an augmented reality environment, and thus the display 32 can be a stereoscopic display that is at least partially see-through, configured to visually augment the appearance of a physical environment viewed by a user through the display 32. In some examples, the display 32 can include one or more regions that are transparent (e.g., optically transparent), and can include one or more regions that are opaque or semi-transparent. In other examples, the display 32 can be transparent (e.g., optically transparent) across the entire available display surface of the display 32. Alternatively, the head-mounted display device 10 can be configured in a fully virtual reality configuration, in which the display 32 is opaque. In the fully virtual reality configuration, a virtual environment can be displayed on the display 32.

[0025] The head-mounted display device 10 can include a processor 12 and a memory 14. In some embodiments, some of the functionality of the processor 12 and / or the memory 14 of the head-mounted display device 10 can be performed by one or more remote computing devices 80 with which the head-mounted display device 10 is configured to communicate. As shown in Figure 1 The remote computing devices 80 include a processor 82 and a memory 84. The remote computing devices 80 may, for example, be server computing devices located in a data center.

[0026] The head-mounted display device 10 can include a communication device suite 16, which includes one or more communication devices, which can include one or more receivers 16A and / or one or more transmitters 16B. In embodiments in which the head-mounted display device 10 communicates with the remote computing devices 80, the one or more receivers 16A can be configured to receive data from the remote computing devices 80, and the one or more transmitters 16B can be configured to transmit data to the remote computing devices 80. In some embodiments, the head-mounted display device 10 can communicate with the remote computing devices 80 via a network, which can be a wireless local area network or a wide area network.

[0027] The processor 12 can be configured to output a mixed reality experience for display on the display 32, the mixed reality experience including one or more virtual objects superimposed on the physical environment. In an augmented reality configuration with an at least partially see-through display, the virtual objects are visually superimposed onto the physical environment visible through the display 32 so as to be perceived at various depths and positions. The head-mounted display device 10 can use stereoscopic vision to visually place virtual objects at desired depths by displaying separate images of the virtual objects to the user's two eyes so that the user will perceive the virtual objects to exist at the desired depths and positions. When the head-mounted display device 10 is in an augmented reality configuration, the processor 12 can be configured to transmit instructions to the display 32 to display virtual objects at world-locked positions in the physical environment. By maintaining an apparent position of virtual objects in the physical environment, the position of the virtual objects can be world-locked, which remains substantially constant under changes in the user's viewing distance and perspective. The virtual objects can be displayed on one or more frames that are displayed at a predetermined frequency in time, such as 60 Hz or 120 Hz.

[0028] Alternatively, the head-mounted display device 100 can be configured in a virtual reality configuration to present a fully virtual reality environment, and thus the display 32 can be a non-transparent stereoscopic display. The head-mounted display device 10 can be configured to display a virtual three-dimensional environment to the user via the non-transparent stereoscopic display. In some embodiments, the head-mounted display device 10 can be configured to display a virtual representation, such as a three-dimensional graphical rendering of the physical environment, in front of the user, which can include additional virtual objects.

[0029] The head-mounted display device 10 can also include one or more speakers configured to emit sound. In some embodiments, the head-mounted display device 10 can include at least a left speaker 34A and a right speaker 34B positioned so that, when the head-mounted display device 10 is worn, the left speaker 34A can be located near the user's left ear and the right speaker 34B can be located near the user's right ear. Thus, the left and right speakers 34A, 34B can emit a stereophonic output. The head-mounted display device 10 can also include one or more haptic feedback devices 36 configured to provide haptic output (e.g., vibrations).

[0030] The head-mounted display device 10 can also include one or more input devices, which can include one or more optical sensors. In one example, the head-mounted display device 10 can include an outward-facing optical sensor 22, which can be configured to detect a real-world background from a similar vantage point (e.g., line of sight) that a user observes through the display 32 in an augmented reality configuration. The head-mounted display device 10 can additionally include an inward-facing optical sensor 24, which can be configured to detect a gaze direction of the user's eyes. It will be appreciated that the outward-facing optical sensor 22 and the inward-facing optical sensor 24 can each include one or more component sensors, such as a visible light camera or a depth camera. In some embodiments, depth data captured by a depth camera can be combined with color information captured by a visible light camera into a single image representation that includes both color data and depth data.

[0031] The head-mounted display device 10 can also include a position sensor system, which can include one or more position sensors 26, such as an accelerometer(s), a gyroscope(s), a magnetometer(s), a global positioning system(s), a multi-point tracker(s), an inertial motion unit (IMU), and / or other sensors that output position data as a position, orientation, and / or movement of the relevant sensor. The head-mounted display device 10 can also include one or more microphones 28 configured to collect sound data.

[0032] Optical sensor information received from the one or more optical sensors and / or position data received from the position sensors 26 can be used to assess a position and orientation of a vantage point of the head-mounted display device 10 relative to other environmental objects. For example, simultaneous localization and mapping (SLAM) can be used to determine a position and orientation of the vantage point. In some embodiments, the position and orientation of the vantage point can be characterized with six degrees of freedom (6DOF) (e.g., world space X, Y, Z, pitch, roll, yaw). This vantage point can be referred to as a device pose or device pose vector, and in some cases, a camera pose vector when referring to server-side rendering of content to be displayed from the same vantage point. The position and / or orientation can be determined by the processor 12 of the head-mounted display device 10 and / or the remote computing device 80.

[0033] Furthermore, the optical sensor information and the position sensor information can be used by the processor 12 to perform analyses of the physical environment, such as depth analysis, surface reconstruction, ambient color and lighting analysis, or other appropriate operations. Specifically, the optical and position sensor information can be used to create a virtual model of the physical environment. The virtual model can be used to determine the corresponding world-locked positions of virtual objects in the virtual space and to add additional virtual objects to be displayed to the user at the desired depth and position. In some embodiments, the physical environment can be modeled over multiple frames. In such embodiments, each frame can have associated optical and / or position sensor information. The optical sensors can also be used to identify machine-recognizable visual features in the physical environment and to calculate frame-to-frame relative pose changes of the head-mounted display device 10 within the world space of the virtual model using the relative movement of these features in consecutive frames.

[0034] Figure 2 It shows that it can be used with Figure 1 A method 100 is used in conjunction with a head-mounted display device 10 and one or more remote computing devices 80 to display content at the head-mounted display device 10. Alternatively, method 100 may be used with some other head-mounted display device. In this example, virtual content is displayed on the head-mounted display device 10 in multiple frames, each frame having a corresponding display time. Step 102 of method 100 may be performed at one or more remote computing devices 80. At step 102, method 100 may include generating a rendering time estimate of the device pose for the head-mounted display device at the current display time of the current frame. The rendering time estimate may be generated at least based on device position data for previous frames. As discussed above, the device position data may include information indicating the position and orientation of the head-mounted display device 10 in space and the motion of the head-mounted display device 10. Device position data may be received from the head-mounted display device 10. Therefore, based on sensor data received from the head-mounted display device 10 in previous frames, the device pose may be estimated for the current frame at one or more remote computing devices 80. The previous frame can be the frame immediately preceding the current frame, or there can be one or more intermediate frames between the previous frame and the current frame.

[0035] Step 104 can be performed at the one or more remote computing devices 80. At step 104, the method 100 can also include rendering the displayed content on a virtual rendering surface of the head-mounted display device 10 to obtain a rendered image. The virtual rendering surface can be a representation of a space in the physical environment in which virtual content can be displayed. For example, the virtual rendering surface can be a cube, a rectangular prism, or some other three-dimensional space in which coordinates of one or more virtual objects can be located. The displayed content can be rendered based at least in part on the rendering time estimate determined at step 102. In some embodiments, the rendered image can include respective color information for a plurality of pixels.

[0036] At step 106, the method 100 can also include transmitting the rendered image from the one or more remote computing devices 80 to the head-mounted display device 10.

[0037] Step 108 of the method 100 can be performed at the processor 12 of the head-mounted display device 10. Step 108 can include receiving position data from the one or more position sensors 26. The position data can include data indicative of an orientation, velocity, and / or acceleration of the head-mounted display device 10 as well as its position. At step 110, the method 100 can also include determining, at the processor 12 of the head-mounted display device 10, an updated device pose of the head-mounted display device 10 based on the position data. In some embodiments, the updated device pose can be determined based at least in part on position data from a previous frame and / or one or more other frames that occurred prior to the current frame. The updated device pose is determined at the display time in preparation for display of the rendered image.

[0038] Step 112 of the method 100 can also be performed at the processor 12 of the head-mounted display device 10. At step 110, the method 100 can also include performing a first spatial correction of color information in pixels of the rendered image. The first spatial correction can be performed at least in part by re-projecting the rendered image based on the updated device pose. When the rendered image is re-projected, the position of at least a portion of the rendered image can be repositioned within the virtual rendering surface. The rendered image can be re-projected such that, when the user views the one or more virtual objects from the position and orientation of the updated device pose, the one or more virtual objects appear to be located at their respective world-locked positions. Thus, as the user's head moves, the apparent position of the virtual objects can appear to remain fixed.

[0039] Step 114 can be performed at the display 32 of the head-mounted display device 10. At step 114, the method 100 can also include performing a second spatial correction on color information in pixels of the re-projected rendered image. The second spatial correction can be performed at least in part by applying a wobble to the re-projected rendered image, thereby generating a sequence of wobbled pixel sub-frames for the current frame. The wobble can be applied within the panel of the display 32 rather than at an external wobbler, as discussed in further detail below. By applying a wobble to the re-projected rendered image, the effective resolution of the display 32 can be increased. Although Figure 2 The re-projection and the second spatial correction are shown as separate steps, but step 114 and step 116 can be combined into a single operation in which the rendered image is both re-positioned and wobbled. When combining step 114 and step 116, the combined step can be performed within the panel of the display 32.

[0040] Step 116 can also be performed at the display 32 of the head-mounted display device 10. At step 116, the method 100 can also include displaying the current frame by displaying the sequence of wobbled pixel sub-frames. Thus, the current frame can be displayed such that the virtual content has been corrected for changes in the device pose and such that the effective resolution of the display 32 has been increased via in-panel wobbling.

[0041] Figure 3 The head-mounted display device 10 and the remote computing device 80 in one or more remote computing devices 80 are shown schematically. As shown in the example of FIG. 1, the head-mounted display device 10 includes a display 32, a processor 34, and a memory 36. The remote computing device 80 includes a processor 82 and a memory 84. Figure 3 As shown in the example of FIG. 1, at the display time 46 of the current frame, the processor 82 of the remote computing device 80 can be configured to compute a render time estimate of the estimated device pose 42 of the head-mounted display device 10. The render time estimate of the estimated device pose 42 can be computed based at least in part on the previous frame position data 44 received from the head-mounted display device 10. Based on the estimated device pose 42, the processor 82 can also be configured to render a rendered image 40 including color information 52 for one or more pixels 50. The processor 82 can render the rendered image 40 such that the one or more virtual objects contained in the rendered image 40 are shown from the position and orientation of the head-mounted display device 10 estimated in the render time estimate of the estimated device pose 42.

[0042] The processor 12 of the head-mounted display device 10 can be configured to receive, via a communication device suite 16, a rendered image 40 of the current frame that has already been rendered at a remote computing device 80. The processor 12 can also be configured to receive position data 60 from one or more position sensors 26. The processor 12 can also be configured to determine an updated device pose 62 of the head-mounted display device 10 based on the position data 60, and in some embodiments, also based on previous frame position data 44.

[0043] The processor 12 may also be configured to generate a reprojected rendered image 64, at least in part, from a rendered image 40 reprojected based on the updated device pose 62. Generating the reprojected rendered image 64 may include performing a first spatial correction on the rendered image 40 to determine corrected color information 66 for each pixel 50. The reprojection may be an adjustment of the rendered image 40 to take into account differences in position and / or orientation between the estimated device pose 42 and the updated device pose 62. Figure 4A An exemplary virtual rendering surface 48 is shown, within which a rendered image 40 is reprojected, thereby obtaining a rendered image 64 reprojected at different positions within the virtual rendering surface 48. Figure 4A In the example, the updated device pose 62 indicates that the head-mounted display device 10 is at a lower height than indicated in the estimated device pose 42. Therefore, the reprojected rendered image 64 is displayed at an increased height relative to the rendered image 40.

[0044] Figure 4B It shows that it can be implemented to generate Figure 4A An example of the post-projection process of the reprojected rendered image 64. Figure 4B At the top left corner, the process begins with a server-side rendering phase, where a remote computing device 80 (e.g., a server) receives the latest device pose of the head-mounted display device 10 and stores it in a pose history. Based on this pose history, an algorithm on the remote computing device 80 calculates a rendering-time estimated device pose 42 at display time and renders the scene based on the estimated device pose 42. The scene is rendered by projecting from the estimated device pose 42 and identifying objects in the virtual environment data that are contained within a defined area wider than the field of view of the virtual camera located at the estimated device pose 42 (as shown in the dashed lines). By rendering a wider area, the fact that the viewport may move within a wider area when the device pose is updated is taken into account, as described below. The image virtually captured by the virtual camera oriented according to the estimated device pose 42 is then rendered onto a rendering surface according to a rendering technique, for example, as shown, the rendering surface may be a rendering box (a box-shaped surface with 5 sides) or a rendering wedge (a double-sided surface with a right and left side).

[0045] It will be appreciated that the rendered render surface comprises an array of pixels 50 in each of the one or more component logical surfaces of the render surface. The data size of the render surface is less than the virtual environment data stored at the remote computing device 80. Server-side rendering enables the faster processor 82 and larger memory 84 of the server to be used to store and render scenes that are to be displayed on the head-mounted display device 10, while transmitting these scenes as relatively lightweight rendered render surfaces over a computer network such as the Internet or a local area network to the head-mounted display device 10. The render surface for a frame to be displayed at a display time is transmitted from the remote computing device 80 to the head-mounted display device 10. At the head-mounted display device 10, onboard sensors such as an IMU determine an updated device pose for the device 62, as described above, and also determine a difference between the estimated device pose 42 and the updated device pose 62 (e.g., in the form of a 6DOF pose transform). Next, in a process referred to as late re-projection, the pixel data in the render surface is re-projected within the render surface using a homography that has been computed based on the pose difference. The re-projected pixel data in the render surface is then written to a viewport and displayed on the display 32 associated with the head-mounted display device 10.

[0046] The updated device pose is transmitted back to the remote computing device 80 for inclusion in the pose history and use in subsequent pose predictions. In this way, the final display of the re-projected rendered image 64 can be adjusted based on display time updates to the device pose, which will result in more accurate positioning of the display image on the display 32. This can be particularly important, for example, in the display of world-locked holograms that are considered to appear to be located on real-world objects, as even slight changes in the location of such objects based on pose estimation errors can cause drift or otherwise reduce the illusion that the holograms are world-locked. Although reference is made to a single image and a single display 32, it will be appreciated that the late re-projection process can be repeated for each of the three color channels in a stereoscopic near-eye display for the head-mounted display device 10 and for each of two different near-eye displays. Figure 4B Only a single image and a single display 32 are described, but it will be appreciated that the late re-projection process can be repeated for each of the three color channels in a stereoscopic near-eye display for the head-mounted display device 10 and for each of two different near-eye displays.

[0047] Thus, in some embodiments, the processor 12 can be configured to perform the first spatial correction separately for each of a plurality of color channels of the rendered image 40, as shown in Figure 5 Figure 5 ​As shown in the example, the color information 52 of the rendered image 40 may include a red channel 52A, a green channel 52B, and a blue channel 52C, which respectively include red, green, and blue values ​​for each pixel 50. When the processor 12 performs the first spatial correction, the processor 12 can be configured to generate corrected color information 66 including corrected red channel 66A, corrected green channel 66B, and corrected blue channel 66C. By reprojecting the different color channels included in the color information 52 individually, the processor 12 can take into account the differences in the angle of refraction for different wavelengths of light. Taking into account these wavelength differences when performing the first spatial correction can result in an increase in the effective resolution of the reprojected rendered image 64 compared to the rendered image 40, a phenomenon referred to as super-resolution.

[0048] Back Figure 3 The processor 12 of the head-mounted display device 10 can also be configured to output the reprojected rendered image 64 to the display 32. The display 32 can be configured to apply a second spatial correction to the corrected color information in the pixels 50 of the reprojected rendered image 64 by applying dithering to the reprojected rendered image 64, at least in part. The second spatial correction can be performed at the dithering circuitry 38 included in the display 32. When the second spatial correction is applied, the display 32 can generate a sequence 70 of dithered pixel subframes 72 for the current frame. Each pixel 50 included in the display 32 can include a plurality of subpixels 54. When the display 32 generates the dithered pixel subframes 72, the display 32 can generate corresponding subpixel information 76, using which each subpixel 54 can be displayed in the dithered pixel subframes 72. As discussed in further detail below, the subpixel information 76 for the subpixel 54 can include a subpixel color 77 and / or a spatial offset 78.

[0049] The display 32 can also be configured to display the current frame by displaying a sequence 70 of jagged pixel subframes 72. When displaying sequence 70, each subpixel 54 can be displayed according to its corresponding subpixel data 76 in each jagged pixel subframe 72. In some embodiments, the sequence 70 of jagged pixel subframes 72 can be displayed at a display rate asynchronous to the rendering rate of the reprojected rendered image 64 output by the processor 12.

[0050] exist Figure 6A-6D An exemplary sequence 70 is shown in the figure. Figure 6A-6DIn the example of FIG. 1, display 32 includes a plurality of pixels 50, each pixel including a respective plurality of addressable sub-pixels 54. Shaking circuit 38 can be configured to apply the shake to the re-projected rendered image 64 at least in part by re-addressing one or more sub-pixels 54 of each pixel 50 of the plurality of pixels 50. When shaking circuit 38 re-addresses one or more sub-pixels 54, shaking circuit 38 can modify which sub-pixels 54 are assigned to which pixels 50, and which locations within those pixels 50. In embodiments in which display 32 is an LCoS display, the shake can be applied to the re-projected rendered image 64 without head-mounted display device 10 including an external shaker in addition to the LCoS display. Thus, the size and weight of head-mounted display device 10 can be reduced by using an LCoS display.

[0051] Each shake pixel sub-frame 72 of sequence 70 can have a corresponding spatial offset 78. In some embodiments, as in the example of FIG. 2, each shake pixel sub-frame 72 has a corresponding spatial offset 78 that is a vertical offset of one sub-pixel 54, a horizontal offset of one sub-pixel 54, or a vertical offset of one sub-pixel 54 and a horizontal offset of one sub-pixel 54. Figure 6A-6D In the example of FIG. 1, each shake pixel sub-frame 72 has a corresponding spatial offset 78 relative to the initial shake pixel sub-frame 72 of sequence 70 that is a vertical offset of one sub-pixel 54, a horizontal offset of one sub-pixel 54, or a vertical offset of one sub-pixel 54 and a horizontal offset of one sub-pixel 54. Figure 6A A first pixel 50A formed by a square of four sub-pixels 54 during a first shake pixel sub-frame 72A is shown. In the example of FIG. 2, the first pixel 50A is centered in the first shake pixel sub-frame 72A. Figure 6B In the example of FIG. 2, the plurality of sub-pixels 54 are re-addressed so that in a second shake pixel sub-frame 72B, a second pixel 50B is formed by a square of four sub-pixels 54 that is shifted right by one sub-pixel 54 relative to the first shake pixel sub-frame 72A shown in FIG. 2. Figure 6A In the example of FIG. 2, the plurality of sub-pixels 54 are re-addressed so that in a second shake pixel sub-frame 72B, a second pixel 50B is formed by a square of four sub-pixels 54 that is shifted right by one sub-pixel 54 relative to the first shake pixel sub-frame 72A shown in FIG. 2. Figure 6C A third shake pixel sub-frame 72C is shown in which the plurality of sub-pixels 54 are re-addressed so that a third pixel 50C is formed by a square of sub-pixels 54 that is shifted down by one sub-pixel 54 relative to the second shake pixel sub-frame 72B. Figure 6D A fourth shake pixel sub-frame 72D is shown in which the plurality of sub-pixels 54 are re-addressed so that a fourth pixel 50D is formed by a square of sub-pixels 54 that is shifted left by one sub-pixel 54 relative to the third shake pixel sub-frame 72C. Following the fourth shake pixel sub-frame 72D, shaking circuit 38 can return to the pixel addressing configuration of the first shake pixel sub-frame 72A.

[0052] In the example of FIG. 1, display 32 includes a plurality of pixels 50, each pixel including a respective plurality of addressable sub-pixels 54. Shaking circuit 38 can be configured to apply the shake to the re-projected rendered image 64 at least in part by re-addressing one or more sub-pixels 54 of each pixel 50 of the plurality of pixels 50. When shaking circuit 38 re-addresses one or more sub-pixels 54, shaking circuit 38 can modify which sub-pixels 54 are assigned to which pixels 50, and which locations within those pixels 50. In embodiments in which display 32 is an LCoS display, the shake can be applied to the re-projected rendered image 64 without head-mounted display device 10 including an external shaker in addition to the LCoS display. Thus, the size and weight of head-mounted display device 10 can be reduced by using an LCoS display. Figure 6A-6DAs shown, in the sequence 70 of the jigged pixel subframes 72, the jigging circuit 38 can modify the corresponding spatial offset 78 of the jigged pixel subframes 72 to move each pixel 50 in a cycle at four different positions. The spatial offset 78 applied to the pixel 50 can effectively increase the resolution of the display 32 by four times. The display 32 can therefore display the reprojected rendered image 64 as if the number of pixels 50 included in the display 32 were four times the original, but without causing an increase in projector size or power consumption, which would result from a fourfold increase in the number of pixels 50 included in the display 32. In addition, the super-resolution achieved as a result of jigging can be independent of the super-resolution achieved as a result of individually reprojecting the color channels of the rendered image 40. Therefore, in Figure 6A-6D The super-resolution implemented in the example can still be implemented in embodiments where the corrected color information 66 includes multiple individually reprojected color channels.

[0053] Back Figure 3 In some embodiments, each shaky pixel subframe 72 included in sequence 70 may have a corresponding subpixel color 77, in addition to having a corresponding spatial offset 78. For example, each subpixel color 77 may be selected from a group consisting of red, green, and blue. In such an embodiment, sequence 70 of shaky pixel subframes 72 may include one or more red shaky pixel subframes, one or more green shaky pixel subframes, and one or more blue shaky pixel subframes. Figure 7A The sequence 170 shows twelve shaky pixel subframes 172A-172L. Figure 7A Sequence 170 loops through sub-pixel color 77 and spatial offset 78, such that each sub-pixel color 77 selected from red, green, and blue is displayed. Figure 6A-6D At each of the four spatial offsets 78 shown. In sequence 170, the first shaky pixel subframe 172A, the fourth shaky pixel subframe 172D, the seventh shaky pixel subframe 172G, and the tenth shaky pixel subframe 172J are red; the second shaky pixel subframe 172B, the fifth shaky pixel subframe 172E, the eighth shaky pixel subframe 172H, and the eleventh shaky pixel subframe 172K are green; and the third shaky pixel subframe 172C, the sixth shaky pixel subframe 172F, the ninth shaky pixel subframe 172I, and the twelfth shaky pixel subframe 172L are blue.

[0054] In other embodiments, such as in Figure 7B As shown, the shaking circuit 38 can be configured to generate a sequence 270, wherein the number of one or more red shaking pixel subframes, one or more green shaking pixel subframes, and one or more blue shaking pixel subframes included in the sequence 270 are not all equal to each other.Figure 7B The shaky pixel subframes 72 included in sequence 270 each have a similarity to... Figure 7A The sequence 170 contains the same corresponding sub-pixel colors 77 and spatial offsets 78, except for the sixth and twelfth wobbly pixel subframes 272F and 272L. The sixth wobbly pixel subframe 272F is green instead of blue, and the twelfth wobbly pixel subframe 272L is red instead of blue. For example, the number of wobbly pixel subframes 72 in sequence 70 with each sub-pixel color 77 can be selected to match the sensitivity of the human eye to different colors of light. In some embodiments, the ratio of wobbly pixel subframes 72 to each sub-pixel color 77 can be dynamically adjusted based on illumination conditions detected by the externally facing optical sensor 22 and / or the internally facing optical sensor 24 of the head-mounted display device 10.

[0055] Figure 8A-8B An exemplary architecture of a projector 300, which may be included in a display 32 of a head-mounted display device according to an exemplary embodiment, is shown. The exemplary projector 300 may include an illumination space 310, a non-emissive display 320, an imaging space 330, and a user's eyes 340. (As shown in...) Figure 8A As shown, the first lighting source 312A and the second lighting source 312B can be located in the lighting space 310, with a first displacement 304 therebetween.

[0056] Light emitted from the first illumination source 312A and the second illumination source 312B can pass through a condenser lens 314, one or more beam-shaping optical elements 316, and a homogenizing lens 318. The condenser lens 314 can form corresponding converged beams from the light emitted from the first illumination source 312A and the second illumination source 312B. Figure 8A-8B In this example, the first illumination source is positioned along the central axis 302 of the condenser lens 314 and the homogenizing lens 318. One or more beam-shaping optical elements 316 may be microlens arrays, as in... Figure 8A The example shown may alternatively include a filter diffuser with a polarization restoration element.

[0057] Light passing through the homogenizing lens 318 can pass through the non-emissive display 320. For example, the non-emissive display 320 can be an LCoS display, a liquid crystal display (LCD), or a digital light processing (DLP) display. The term "non-emissive display" is a term in the art referring to a display in which light is generated behind the user-facing display surface and an image is formed by filtering that light at a non-emissive display. Therefore, the non-emissive display receives backlight and transmits filtered light. In such a display, since the light source is not within the display itself but in the backlight, the display is called non-emissive. As in Figure 8AAs shown in the example projector 300 in FIG. 3, the light emitted by the second illumination source 312B can have an angular offset with respect to the light emitted by the first illumination source 312A as it passes through the non-emissive display 320.

[0058] Figure 8B An imaging space 330 and an eye 340 of a user are shown in accordance with an example of the present disclosure. Figure 8A The imaging space 330 can include a collimating lens 332 and a pupil replication waveguide 334. In some embodiments, as shown in FIG. 3, the collimating lens 332 and the pupil replication waveguide 334 can be offset from the central axis 302 of the condenser lens 314 and the homogenizing lens 318 by a second displacement 306. The collimating lens 332 can be configured to receive the light that passes through the non-emissive display 320 and focus the light onto the pupil replication waveguide 334. Figure 8B

[0059] The pupil replication waveguide 334 can include a first input grating 336A, a second input grating 336B, a redirecting grating 338, and an extraction grating 339. The first input grating 336A and the second input grating 336B can be first and second entrance ports for respective sub-pixels 54, where the first input grating 336A is configured to receive light that is initially output by the first illumination source 312A and the second input grating 336B is configured to receive light that is initially output by the second illumination source 312B. The first input grating 336A and the second input grating 336B can each have a respective grating period. The grating periods of the first input grating 336A and the second input grating 336B can be different from one another, such that the first input grating 336A has a first coupling angle and the second input grating has a second coupling angle. Thus, the light received at the first input grating 336A and the light received at the second input grating 336B can pass through their respective gratings at different angles, as discussed in further detail below.

[0060] After passing through the first input grating 336A and the second input grating 336B, the light can pass through the redirecting grating 338 and the extraction grating 339 to reach the eye 340 of the user. In Figure 8A-8B In embodiments of the present disclosure, a single redirecting grating 338 and a single extraction grating 339 are used for both the first input grating 336A and the second input grating 336B. In other embodiments, separate redirecting gratings 338 and / or separate extraction gratings 339 can be used for the first input grating 336A and the second input grating 336B. Additionally or alternatively, in some embodiments, the projector 300 can include a waveguide having some other architecture. For example, the waveguide can include one or more prisms, one or more cascaded half-tone mirrors, or one or more pin light mirrors. In other embodiments, the waveguide can include one or more exit pupils without including an extraction grating 339.​

[0061] The lens 342 of the user’s eye 340 can focus the light received from the pupil replication waveguide 334 onto the user’s retina to form first and second sub-pixel images 344A and 344B that are spatially offset from each other. Thus, due to the first displacement 304 between the first and second illumination sources 312A and 312B, the first and second sub-pixel images 334A and 344B can be displayed at different locations in the user’s field of view.

[0062] Figure 9A A top view of the pupil replication waveguide 334 shown in Figure 8B is shown. Additionally, Figure 9B A cross-sectional view of the pupil replication waveguide 334 is shown. As shown in Figure 9B , the light that has passed through the first and second input gratings 336A and 336B has been redirected via total internal reflection within the redirecting grating 338 to impinge on the extraction grating 339. Because the first and second input gratings 336A and 336B have different coupling angles in the example shown, Figure 9A-9B the light that passes through the second input grating 336B passes through the redirecting grating 338 at a different angle than the light that passes through the first input grating 336A. Figure 9B The dashed line in shows the path that the light that passes through the second input grating 336B would take if the first and second input gratings 336A and 336B had the same coupling angle. The point at which the light that passes through the second input grating 336B impinges on the extraction grating 339 is also spatially offset due to the difference between the first and second coupling angles. Additionally, because the light that passes through the first input grating 336A and the light that passes through the second input grating 336B impinge on the extraction grating 339 at different angles, it is also output to the user’s eye 340 at different angles. Thus, the first and second sub-pixel images 344A and 344B can be displayed at different locations.

[0063] Figure 9C A top view of an alternative configuration of a pupil replication waveguide 354 is shown. Figure 9C The configuration of shows a first redirecting grating 338A and a second redirecting grating 338B that can be configured to redirect the light that passes through the first and second input gratings 336A and 336B, respectively. In the example shown, Figure 9C in the embodiment of, the first and second redirecting gratings 338A and 338B both redirect the light to the same extraction grating 339. In other embodiments, the first and second redirecting gratings 338A and 338B can redirect the light to separate extraction gratings 339.

[0064] AlthoughFigure 8A Two illumination sources 312A and 312B are shown and 8B-9C shows two input gratings 336A and 336B, but each pixel 50 of the display 32 can be formed using four illumination sources and four input gratings corresponding to the four sub-pixels 54. In such embodiments, the third and fourth illumination sources can be positioned relative to the first and second illumination sources 312A and 312B in a manner similar to that shown in Figure 8A The first illumination source 312A and the second illumination source 312B of FIG. 8B-9C are positioned in a direction orthogonal to the page, thereby forming a rectangle of illumination sources. Multiple of these rectangular pixels 50 can be arranged in a rectangular grid, as shown in Figure 6A-6D FIG. 8B-10A. Similarly, the input gratings contained in the pupil replication waveguide 334 can be arranged in a rectangular grid to form pixels 50 each having four corresponding input gratings.

[0065] Figure 10A-10B Another exemplary embodiment of a pupil replication waveguide 434 is shown in a top view in Figure 10A and a cross-sectional view in Figure 10B The pupil replication waveguide 434 can include an input pupil 436, a redirecting grating 438, and an extraction grating 439. In contrast to the embodiments of Figure 8A-9B and Figure 9C The redirecting grating 438 includes multiple embedded gratings. Figure 10B A first embedded grating 437A, a second embedded grating 437B, a third embedded grating 437C, and a fourth embedded grating 437D are shown stacked within the redirecting grating 438 below the input pupil 436. The first embedded grating 437A, the second embedded grating 437B, the third embedded grating 437C, and the fourth embedded grating 437D can have respective coupling angles that are different from one another, such that each of the embedded gratings 437A, 437B, 437C, and 437D corresponds to a different sub-pixel location. In Figure 10B these different coupling angles are illustrated by the diverging paths of the emitted light from the extraction grating 439.

[0066] Each of the embedded gratings 437A, 437B, 437C, and 437D can be configured to be electronically activated and deactivated. The embedded gratings 437A, 437B, 437C, and 437D can each be configured to refract incident light when activated and to transmit incident light without refracting the incident light when not activated. Thus, the wobble circuit 38 can be configured to apply the wobble to the re-projected rendered image 64 by activating and deactivating the embedded gratings 437A, 437B, 437C, and 437D to apply the respective spatial offsets 78 to the plurality of pixels 50. In some embodiments, for each pixel 50, the wobble circuit 38 can be configured to activate at most one of the embedded gratings 437A, 437B, 437C, or 437D at any given wobble pixel subframe 72. In other embodiments, an embedded grating of the plurality of embedded gratings 437A, 437B, 437C, and 437D can remain activated while one or more of the other embedded gratings are activated.

[0067] As an alternative to including multiple embedded gratings in the redirecting grating of a single pupil replication waveguide 434, the projector 300 instead includes multiple spatially multiplexed pupil replication waveguides that include respective input gratings configured to be electronically activated and deactivated, such that each input grating is configured to refract incident light when activated and to transmit without refracting the incident light when not activated. Each of these spatially multiplexed pupil replication waveguides can include one or more input gratings. From this, the electronic activation and deactivation of the input gratings can allow for temporal multiplexing as well as spatial multiplexing of the input gratings. In such embodiments, the spatial offsets 78 of the sub-pixels 54 can be applied by addressing the input gratings, as indicated by the sequence 70 of wobble pixel subframes 72.

[0068] In some embodiments, image warping can be applied to the re-projected rendered image 64 in conjunction with the wobble. In such embodiments, the sub-pixel information 76 of the pixels 50 in the wobble pixel subframes 72 can include respective spatial offsets 78 of more than one sub-pixel 54. Each of these spatial offsets 78 can be vertical, horizontal, or both vertical and horizontal. As some particular examples, the sub-pixels 54 in the wobble pixel subframes 72 can be spatially offset by a distance of 50 pixels 50 and one sub-pixel 54, 100 pixels 50 and one sub-pixel 54, or 200 pixels 50 and one sub-pixel 54. Other pixel dimension offsets are also contemplated. Using these spatial offsets 78 of more than one sub-pixel 54 distance, field of view (FOV) stitching can be performed, where different regions of the re-projected rendered image 64 are displayed in different wobble pixel subframes 72.

[0069] Figure 11 Examples are shown of a first pixel layout 510 in which jitter is performed without FOV stitching, a second pixel layout 520 in which FOV stitching is performed without jitter, and a third pixel layout 530 in which both jitter and FOV stitching are performed. In the first pixel layout 510, spatial offsets 78 to the downside of one subpixel 54 and to the right of one subpixel 54 are shown between two jitter pixel subframes 72, as in Figure 6A-6D As shown in the example.

[0070] The second pixel layout 520 shows four FOV regions 522 that can be reused spatially and / or temporally. Subpixels 54 contained within the currently displayed FOV region 522 are shown as fill in the second pixel layout 520, while subpixels 54 of the FOV region 522 displayed at other times are shown as outlines. Figure 11 In the example, the FOV regions 522 do not overlap. In an embodiment where the FOV regions 522 are spatially multiplexed, the FOV regions 522 can be projected onto the user's eye 340 via separate optical waveguide replication. In an embodiment where the FOV regions 522 are temporally multiplexed, different FOV regions 522 can be displayed in separate subframes. When the display 32 of the head-mounted display device 10 includes Figure 8A-9B Pupil replication waveguide 334, Figure 9C Pupil replication waveguide 354 or Figure 10A-10B When the pupil replication waveguide 434 is used, FOV stitching can be performed by applying angular displacement to the light leaving the extraction grating of the pupil replication waveguide. Although in Figure 11 The example shows four FOV regions 522, but other embodiments may use a different number of FOV regions 522.

[0071] exist Figure 11 The third pixel layout 530 shown includes four wobbly FOV regions 532 that can be displayed in individual wobbly pixel subframes 72. The four wobbly FOV regions 532 shown in the third pixel layout 530 overlap, such that the subpixel density of the third pixel layout is higher near the center than at the edges. This increase in density near the center of the third pixel layout 530 can provide increased resolution near the fovea at the center of the user's retina, where the perceived sharpness of the user's vision is higher than at the periphery of the user's vision. The location of the fovea can be determined using image data collected by the inward-facing optical sensor 24 via gaze tracking. Therefore, a non-uniform subpixel density that appears sharper can be used to display the current frame generated from the reprojected rendered image 64 without increasing the number of subpixels 54 included in the current frame.

[0072] Similar to FOV region 522 in second pixel layout 520, shake FOV region 532 can be spatially and / or temporally multiplexed. In embodiments in which shake FOV region 532 is spatially multiplexed, shake FOV region 532 can be output to user's eye 340 through separate pupil replication waveguides. In such embodiments, shaking can be performed at each of the pupil replication waveguides.

[0073] The separate pupil replication waveguides for shake FOV region 532 can have different angular bandwidths over which light can be emitted from their respective output gratings. By using separate angular bandwidths for the pupil replication waveguides, the coupling efficiency of the pupil replication waveguides can be increased since the extraction gratings emit light over a smaller range of angles. Additionally, when the pupil replication waveguides have different angular bandwidths, each shake FOV region 532 can be more uniformly illuminated.

[0074] Using the systems and methods discussed above, the size and power consumption of a near-eye display included in a head-mounted display device can be reduced. Applying shaking to a re-projected rendered image can allow the apparent resolution of the display perceived by a user to be increased, for example, by a factor of four. Additionally, super-resolution can be achieved by performing post-reprojection on a rendered image and by applying shaking to the re-projected rendered image, thereby allowing further increases in perceived resolution. Thus, the techniques discussed above can allow smaller, less power-intensive projectors to be used without the user perceiving a reduction in display resolution. Size savings can also be achieved using the in-panel shaking techniques discussed above rather than including an external shaker in the head-mounted display device.

[0075] In some embodiments, the methods and processes described herein can be associated with a computing system of one or more computing devices. Specifically, such methods and processes can be implemented as a computer-application or service, an application-programming interface (API), a library, and / or other computer-program product.

[0076] Figure 12 A non-limiting embodiment of a computing system 600 is schematically illustrated that is capable of carrying out one or more of the methods and processes described above. The computing system 600 is shown in simplified form. The computing system 600 can embody the head-mounted display device 10 and the remote computing device 80 described above and illustrated in FIG. 1. Figure 1 The computing system 600 can take the form of one or more personal computers, server computers, tablet computer(s), home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone(s)), and / or other computing devices and wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0077] Computing system 600 includes a logic processor 602, volatile memory 604, and a non-volatile storage device 606. Computing system 600 can optionally include a display subsystem 608, input subsystem 610, communication subsystem 612, and / or other components not shown in FIG. 6. Figure 12

[0078] Logic processor 602 includes one or more physical devices configured to execute instructions. For example, the logic processor can be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise arrive at desired results.

[0079] The logic processor can include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logic processor can include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. Processors of the logic processor 602 can be single-core or multi-core, and the instructions executed thereon can be configured for sequential, parallel, and / or distributed processing. Individual components of the logic processor optionally can be distributed among two or more separate devices, which can be remotely located and / or configured for coordinated processing. Aspects of the logic processor can be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration. In such a scenario, these virtualized aspects are run on different physical logic processors of various different machines, which will be appreciated by those skilled in the art.

[0080] Non-volatile storage device 606 includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device 606 can be transformed in accordance with the implementation.

[0081] ​The non-volatile storage device 606 can include physical devices such as optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.) and / or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 606 can include nonvolatile, dynamic, static, read / write, read-only, sequential-access, location- addressable, file-addressable, and / or content-addressable devices. It will be appreciated that the non-volatile storage device 606 is configured to retain instructions even when power is cut to the non-volatile storage device 606.

[0082] The volatile memory 604 can include physical devices such as random access memory (RAM). The volatile memory 604 is typically used by the logic processor 602 to temporarily store information during processing of software instructions. It will be appreciated that the volatile memory 604 does not continue to store instructions when power is cut to the volatile memory 604.

[0083] Aspects of the logic processor 602, the volatile memory 604, and the non-volatile storage device 606 can be integrated into one or more hardware-logic components. Such hardware-logic components can include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.

[0084] The terms "module," "program," and "engine" can be used to describe an aspect of the computing system 600 typically implemented in software by a processor to perform a particular functionality tied to a specific logic that was configured at the time of manufacture of the processor. Accordingly, a module, program, or engine can be instantiated via logic processor 602 executing an instance of the program to perform the functionality associated with the module, program, or engine. Different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine can be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0085] When included, display subsystem 608 can be used to present a visual representation of data held by non-volatile storage device 606. This visual representation can take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of display subsystem 608 can likewise be transformed to visually represent changes in the underlying data. Display subsystem 608 can include one or more display devices utilizing virtually any type of technology. Such display devices can be combined with logic processor 602, volatile memory 604, and / or non-volatile storage device 606 in a shared enclosure, or such display devices can be peripheral display devices.

[0086] When included, input subsystem 610 can comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem can comprise or interface with selected natural user input (NUI) componentry. Such componentry can be integrated or peripheral, and the transduction and / or processing of input actions can be handled on- or off-board. Example NUI componentry can include a microphone for speech and / or voice recognition; an infrared, color, stereoscopic, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; as well as electric-field sensing componentry for

[0087] When included, communication subsystem 612 can be configured to communicatively couple various computing devices described herein to each other, and to other devices. Communication subsystem 612 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network, such as an HDMI connection over Wi-Fi. In some embodiments, the communication subsystem can allow computing system 600 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0088] The following paragraphs describe several aspects of the present disclosure. A head-mounted display device is provided that includes one or more position sensors and a processor. The processor can be configured to receive a rendered image of a current frame that has been rendered at a remote computing device. The processor is further configured to receive position data from the one or more position sensors. The processor can be further configured to determine an updated device pose of the head-mounted display device based on the position data. The processor can be further configured to apply a first spatial correction to color information in pixels of the rendered image at least in part by reprojecting the rendered image based on the updated device pose. The head-mounted display device can further include a display configured to apply a second spatial correction to the color information in the pixels of the reprojected rendered image at least in part by applying a wobble to the reprojected rendered image, thereby generating a sequence of wobbled pixel subframes for the current frame. The display can be further configured to display the current frame by displaying the sequence of wobbled pixel subframes.

[0089] According to this aspect, the wobble is applied to the reprojected rendered image at a wobble circuit included in the display.

[0090] According to this aspect, the display can include a plurality of pixels, each pixel of the plurality of pixels including a respective plurality of addressable subpixels. The wobble circuit can be configured to apply the wobble to the reprojected rendered image at least in part by readdressing one or more subpixels of each pixel of the plurality of pixels.

[0091] According to this aspect, each wobbled pixel subframe has a corresponding subpixel color and a corresponding spatial offset.

[0092] According to this aspect, the sequence of wobbled pixel subframes can include one or more red wobbled pixel subframes, one or more green wobbled pixel subframes, and one or more blue wobbled pixel subframes. The number of one or more red wobbled pixel subframes, the number of one or more green wobbled pixel subframes, and the number of one or more blue wobbled pixel subframes included in the sequence are not all equal to each other.

[0093] According to this aspect, the corresponding spatial offset of each wobbled pixel subframe can be a vertical offset of one subpixel, a horizontal offset of one subpixel, or a vertical offset of one subpixel and a horizontal offset of one subpixel.

[0094] According to this aspect, the subpixels of the current frame can be displayed at a non-uniform subpixel density.

[0095] According to this aspect, the display can be a liquid crystal on silicon (LCoS) display.

[0096] According to this aspect, the rendered image can be rendered at the remote computing device based on a rendering time estimate of an estimated device pose of the head-mounted display device at a display time of the current frame.

[0097] According to this aspect, the display can further include a pupil replication waveguide including a first entrance aperture having a first coupling angle and a second entrance aperture having a second coupling angle. The first coupling angle can be different from the second coupling angle.

[0098] According to another aspect of the disclosure, a method for use with a head-mounted display device is provided. The method can include receiving, at a processor, a rendered image of a current frame that has been rendered at a remote computing device. The method can further include receiving, at the processor, position data from one or more position sensors. The method can further include determining, at the processor, an updated device pose of the head-mounted display device based on the position data. The method can further include applying, at the processor, a first spatial correction to color information in pixels of the rendered image at least in part by reprojecting the rendered image based on the updated device pose. The method can further include, at a display: applying a second spatial correction to the color information in the pixels of the reprojected rendered image at least in part by applying a dither to the reprojected rendered image, thereby generating a sequence of dithered pixel subframes for the current frame. The method can further include displaying, at the display, the current frame by displaying the sequence of dithered pixel subframes.

[0099] According to this aspect, the dither can be applied to the reprojected rendered image at dithering circuitry included in the display.

[0100] According to this aspect, the display can include a plurality of pixels, each pixel of the plurality of pixels including a respective plurality of addressable subpixels. Applying a dither to the reprojected rendered image can include readdressing one or more subpixels of each pixel of the plurality of pixels.

[0101] According to this aspect, each dithered pixel subframe has a corresponding subpixel color and a corresponding spatial offset.

[0102] According to this aspect, the sequence of dithered pixel subframes can include one or more red dithered pixel subframes, one or more green dithered pixel subframes, and one or more blue dithered pixel subframes. The number of the one or more red dithered pixel subframes, the number of the one or more green dithered pixel subframes, and the number of the one or more blue dithered pixel subframes included in the sequence are not all equal to one another.

[0103] According to this aspect, the corresponding spatial offset of each dithered pixel sub-frame is a vertical offset of one sub-pixel, a horizontal offset of one sub-pixel, or a vertical offset of one sub-pixel and a horizontal offset of one sub-pixel.

[0104] According to this aspect, the rendered image can be rendered at a remote computing device based on a rendering time estimate of an estimated device pose of the head-mounted display device at a display time of the current frame.

[0105] According to another aspect of the disclosure, a head-mounted display device is provided that includes one or more position sensors. The head-mounted display device can also include a liquid crystal on silicon (LCoS) display that includes a plurality of pixels, each pixel of the plurality of pixels including a respective plurality of addressable sub-pixels. The head-mounted display device can also include a processor configured to receive a rendered image of a current frame that has been rendered at a remote computing device. The processor can also be configured to receive position data from the one or more position sensors. The processor can also be configured to determine an updated device pose of the head-mounted display device based on the position data. The processor can also be configured to apply a first spatial correction to color information in pixels of the rendered image based at least in part by re-projecting the rendered image based on the updated device pose. The LCoS display can also be configured to apply a second spatial correction to the color information in the pixels of the re-projected rendered image at a dithering circuit by re-addressing one or more sub-pixels of the re-projected rendered image, thereby generating a sequence of dithered pixel sub-frames for the current frame. The LCoS display can also be configured to display the current frame by displaying the sequence of dithered pixel sub-frames.

[0106] According to this aspect, each dithered pixel sub-frame can have a corresponding sub-pixel color selected from a group consisting of red, green, and blue. Each dithered pixel sub-frame can have a corresponding spatial offset. Each spatial offset can be a vertical offset of one sub-pixel, a horizontal offset of one sub-pixel, or a vertical offset of one sub-pixel and a horizontal offset of one sub-pixel.

[0107] According to this aspect, the display can also include a pupil replication waveguide that includes a first entrance aperture having a first coupling angle and a second entrance aperture having a second coupling angle. The first coupling angle can be different than the second coupling angle.

[0108] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. As such, various acts illustrated and / or described can be performed in the sequence illustrated and / or described, in other sequences, concurrently, at different times, or omitted. Likewise, the order of the above-described processes can be changed.

[0109] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts and / or attributes disclosed herein, as well as any and all equivalents thereof.

Claims

1. A head-mounted display device, comprising: one or more position sensors; a processor configured to: receive a rendered image of a current frame that has been rendered at a remote computing device; receive position data from the one or more position sensors; determine an updated device pose of the head-mounted display device based on the position data; and apply a first spatial correction to color information in pixels of the rendered image at least in part by reprojecting the rendered image based on the updated device pose; and a display configured to: apply a second spatial correction to the color information in the pixels of the reprojected rendered image at least in part by applying a wobble to the reprojected rendered image, thereby generating a sequence of wobbled-pixel subframes for the current frame, wherein the sequence of wobbled-pixel subframes includes a plurality of wobbled-pixel subframes; and display the current frame by displaying the sequence of wobbled-pixel subframes, wherein the wobble is applied to the reprojected rendered image at a wobble circuit contained in the display.

2. The head-mounted display device of claim 1, wherein: the display includes a plurality of pixels, each pixel including a respective plurality of addressable subpixels; and the wobble circuit is configured to apply the wobble to the reprojected rendered image at least in part by readdressing one or more subpixels of each pixel of the plurality of pixels. Each wobbled-pixel subframe has a corresponding subpixel color and a corresponding spatial offset.

3. The head-mounted display device of claim 2, wherein, 4. The head-mounted display device of claim 3, wherein: the sequence of wobbled-pixel subframes includes one or more red wobbled-pixel subframes, one or more green wobbled-pixel subframes, and one or more blue wobbled-pixel subframes; and the number of the one or more red wobbled-pixel subframes, the number of the one or more green wobbled-pixel subframes, and the number of the one or more blue wobbled-pixel subframes included in the sequence are not all equal to one another. The corresponding spatial offset of each wobbled-pixel subframe is a vertical offset of one subpixel, a horizontal offset of one subpixel, or a vertical offset of one subpixel and a horizontal offset of one subpixel.

5. The head-mounted display device of claim 3, wherein, Subpixels of the current frame are displayed at a non-uniform subpixel density.

6. The head-mounted display device of claim 2, wherein, The display is a liquid crystal on silicon (LCoS) display.

7. The head-mounted display device of claim 1, wherein, The rendered image is rendered at the remote computing device according to a rendering-time estimate of an estimated device pose of the head-mounted display device at a display time of the current frame.

8. The head-mounted display device of claim 1, wherein, 9. The head-mounted display device of claim 1, wherein: the display further includes a pupil replication waveguide, the pupil replication waveguide including a first entrance aperture having a first coupling angle and a second entrance aperture having a second coupling angle; and the first coupling angle is different from the second coupling angle.

10. A method for use with a head-mounted display device, the method comprising: at a processor: receiving a rendered image of a current frame that has been rendered at a remote computing device; receiving position data from one or more position sensors; ​ determine an updated device pose of the head-mounted display device based on the position data; and apply a first spatial correction to color information in pixels of the rendered image at least in part by re-projecting the rendered image based on the updated device pose; and at a display: apply a second spatial correction to the color information in the pixels of the re-projected rendered image at least in part by applying a wobble to the re-projected rendered image, thereby generating a sequence of wobbled pixel sub-frames for the current frame, wherein the sequence of wobbled pixel sub-frames comprises a plurality of wobbled pixel sub-frames; and display the current frame by displaying the sequence of wobbled pixel sub-frames, wherein the wobble is applied to the re-projected rendered image at a wobble circuit included in the display.

11. The method of claim 10, wherein: the display comprises a plurality of pixels, each pixel comprising a respective plurality of addressable sub-pixels; and applying the wobble to the re-projected rendered image comprises re-addressing one or more sub-pixels of each pixel of the plurality of pixels.

12. The method of claim 11, wherein, Each wobbled pixel sub-frame has a corresponding sub-pixel color and a corresponding spatial offset.

13. The method of claim 12, wherein, The corresponding spatial offset of each wobbled pixel sub-frame is a vertical offset of one sub-pixel, a horizontal offset of one sub-pixel, or a vertical offset of one sub-pixel and a horizontal offset of one sub-pixel.

Images