Rendered optical super-resolution display

Abstract

A method includes rendering, by at least one processor, a first subframe (400) of an image, where the first subframe (400) includes a first subset of pixels (202) of the image. The method includes displaying the first subframe (400) on a display. The method further includes rendering, by the at least one processor, a second subframe of the image, where the second subframe includes a second subset of pixels (212) of the image, and where the second subframe is diagonally shifted from the first subframe (400) by half a pixel. The method further includes displaying the second subframe on the display after displaying the first subframe (400), where the display is diagonally optically shifted by half a pixel to display the second subframe.

Description

BACKGROUND

[0001] Augmented reality (AR) and virtual reality (VR) systems provide simulated experiences that are similar or different from the real world. VR systems can be used for entertainment or educational purposes. VR systems use VR headsets or multi-projection environments to generate realistic images, sounds, and other sensations to simulate a user's physical presence in a virtual environment. AR is an interactive experience of a real-world environment where the objects that exist in the real world are enhanced by computer-generated perceptual information, sometimes across multiple human senses, such as visual, auditory, or tactile. SUMMARY

[0002] In some examples, a method includes rendering, by at least one processor, a first subframe of an image, where the first subframe includes a first subset of pixels of the image. The method includes displaying the first subframe on a display. The method also includes rendering, by the at least one processor, a second subframe of the image, where the second subframe includes a second subset of pixels of the image, and where the second subframe is diagonally shifted by half a pixel from the first subframe. The method also includes displaying the second subframe on the display after displaying the first subframe, where the display is diagonally optically shifted by half a pixel to display the second subframe.

[0003] In some examples, a system includes a processing unit configured to render a first subframe of an image, where the first subframe includes a first subset of pixels of the image. The processing unit is also configured to apply an anti-aliasing filter to the first subset of pixels. The processing unit is configured to transmit the first subframe. The processing unit is also configured to render a second subframe of the image, where the second subframe includes a second subset of pixels of the image, and where the second subframe is diagonally shifted by half a pixel from the first subframe. The processing unit is also configured to apply the anti-aliasing filter to the second subset of pixels and transmit the second subframe.

[0004] In some examples, a system includes a headset including a display and a graphics processing unit (GPU). The GPU is configured to render a first subframe of an image, where the first subframe includes a first subset of pixels of the image. The GPU is also configured to render a second subframe of the image, where the second subframe includes a second subset of pixels of the image, where the second subframe is diagonally shifted by half a pixel from the first subframe. The headset also includes a spatial light modulator (SLM). The SLM is configured to receive the first subframe from the GPU and project the first subframe to the display. The SLM is also configured to receive the second subframe from the GPU and project the second subframe to the display, where the display is diagonally optically shifted by half a pixel to display the second subframe relative to displaying the first subframe. BRIEF DESCRIPTION OF DRAWINGS

[0005] For a detailed description of various examples, reference will now be made to the accompanying drawings in which:

[0006] Figure 1 is a schematic diagram of an AR system in various examples.

[0007] Figure 2 is a schematic diagram of a pixel of a rendered frame in various examples.

[0008] Figure 3 is a diamond passband anti-aliasing filter in various examples.

[0009] Figure 4 is a filter span across super-samples in various examples.

[0010] Figure 5 is a filter span across super-samples in various examples.

[0011] Figure 6 is a chart of optical super-resolution results in various examples.

[0012] Figure 7 is a schematic diagram of a pixel of a rendered frame in various examples.

[0013] Figure 8 is a demonstration of pixelated mitigation in various examples.

[0014] Figure 9 is a demonstration of higher frequency reproduction in various examples.

[0015] Figure 10 is a flowchart of a method of optical super-resolution in various examples.

[0016] Figure 11 is a flowchart of a method of optical super-resolution in various examples. DETAILED DESCRIPTION

[0017] AR effects are often produced by head-mounted viewers that include a head-mounted display with a small display screen in front of the eyes. One way to enhance the user experience is to increase the resolution of this display. One way to increase resolution includes adding pixels to the display device, but more pixels increase the size and cost of the system. For AR or VR head-mounted viewers that use a battery, battery life can also be shortened when using a display with a higher number of pixels.

[0018] Examples herein utilize optical super-resolution (OSR) to increase the perceived resolution of a display. A processor, such as a graphics processing unit (GPU), renders image frames that are spatially offset from one another. In one example, a first frame has no offset and a second frame is offset horizontally by 1 / 2 pixel and vertically by 1 / 2 pixel relative to the first frame. The GPU alternates between frames with no offset and frames that have a 1 / 2 pixel by 1 / 2 pixel offset relative to the non-offset frame. Additionally, in some examples, an anti-aliasing scheme is applied to each rendered frame to reduce artifacts associated with downsampling the frames. By rapidly cycling through these spatially offset frames, the perceived resolution of the display by the observer is higher than the native resolution of the display. In other examples, more than two spatially offset frames may be used.

[0019] Figure 1 is a schematic diagram of an AR system in various examples. A head-mounted viewer 100 is a head-mounted device that displays image or video data to a user. The examples herein are described in the context of a head-mounted display, but other examples use non-head-mounted displays. The head-mounted viewer 100 includes a central processing unit (CPU) 102 that provides processing power. The head-mounted viewer 100 also includes a GPU 104 that creates and renders images for display. Memory 106 stores data, such as image and video data. Memory 106 may also store instructions for the CPU 102 and / or GPU 104. Sensors 108 are any sensors for detecting information about the environment, the head-mounted viewer 100, or the user. Examples of sensors 108 include hardware that detects spatial and motion information, such as position, orientation, rotation, velocity, and acceleration. Sensors 108 may also include infrared, pyroelectric, ultrasonic, microphone, laser, optical, capacitive, acoustic, and / or inductive sensors.

[0020] The head mounted viewer 100 includes a spatial light modulator (SLM) 110. One type of SLM in the example is a digital micromirror device (DMD). A DMD is a microelectromechanical system (MEMS) that has an array of hundreds of thousands or millions of micromirrors on its surface. Each micromirror corresponds to a pixel in the image that is projected onto the micromirror and then reflected from the micromirror to the display 112. Other types of SLMs that can be used in some examples are liquid crystal display (LCD) systems or liquid crystal on silicon (LCoS) systems. The display 112 can be any suitable type of viewing surface. In one example, the display 112 can include waveguide optical elements that provide a near-eye display to the head mounted viewer 100. The optical device 114 includes any lenses, prisms, or other optical components used to create an image for display in the head mounted viewer 100. The input / output (I / O) 116 includes any interface for coupling the head mounted viewer 100 to another device, such as a console or computer ( Figure 1The I / O 116 can also include one or more input devices, such as a keyboard, mouse, controller, or any other suitable device for receiving action requests and communicating the received action requests. The communication component 118 includes any component (e.g., a transmitter, a receiver, transceiver, etc.) that enables communication between the head-mounted viewer 100 and one or more communication networks. In some examples, the communication component 118 includes hardware capable of conducting data communications using any of a variety of wireless protocols (e.g., Wi-Fi, Bluetooth, cellular, etc.), wired protocols, and / or any other suitable communication protocol. The head-mounted viewer 100 also includes a bus 120 that connects and allows communication between components of the head-mounted viewer 100. The head-mounted viewer 100 includes an optical engine system 122. In one example, the optical engine system 122 can include an actuator. The actuator displaces or shifts the display a predetermined distance in the vertical and / or horizontal direction per frame. For example, a first frame can have no shift, while a second frame has the display shifted horizontally by 1 / 2 pixel and vertically by 1 / 2 pixel relative to the first frame by the actuator. The GPU alternates between the frame with no shift and the frame with a 1 / 2 pixel by 1 / 2 pixel shift relative to the non-shifted frame, and the actuator shifts the display by the appropriate amount and direction per frame in synchronization with the GPU. If more than two spatially shifted frames are used, the actuator moves the display in the appropriate manner for each frame to display the frame. In another example, an optical element in the head-mounted viewer 100 can be moved instead of the display, such as a mirror or lens. In another example, a glass plate in the optical path in the head-mounted viewer 100 can be shifted by an actuator.

[0021] In one example, the OSR is performed in a closed rendering ecosystem. In a closed ecosystem, the source for display is rendered within a single device, such as the head-mounted viewer 100. The GPU 104 in the head-mounted viewer 100 renders the source for display on the display 112 also in the head-mounted viewer 100. Thus, the GPU 104 is programmed to render the source for the specific hardware within the head-mounted viewer 100. In the example herein, the SLM 110 is a spatial light modulator that includes an array of micromirrors that is 1280 columns by 720 rows in size. The SLM 110 outputs an image that is 1280 pixels by 720 pixels in size for display on the display 112. Other sizes of arrays are used in other examples. Thus, the final image size of the display is 1280x720. However, a higher resolution image is rendered by the GPU 104. In the present example, the GPU 104 renders an image that is 2560 columns by 1440 rows of pixels at 150 Hertz (Hz), which is four times the pixels of the 1280x720 display. From the 2560x1440 rendered image frame, two or more spatially offset frames of size 1280x720 are created and then displayed to the viewer fast enough so that the viewer does not perceive any flicker between the slightly offset frames. Displaying all of the spatially offset frames at a rate of 60 Hz or faster generally prevents the human eye from perceiving flicker because the human eye acts as a temporal low-pass filter and combines the separate frames into one perceived image. The rate of 60 Hz corresponds to 16.67 milliseconds, so both spatially offset frames are displayed within a 16.67 millisecond time window. In the example herein, two spatially offset frames are displayed within a 16.67 millisecond or less time window. If four spatially offset frames are used in another example, all four spatially offset frames are displayed within a 16.67 millisecond time window.

[0022] In one example, the 1280x720 spatially offset frames are created by subsampling the 2560x1440 supersample pixels rendered by the GPU 104. The 1280x7200 pixels selected for the subframes are arranged in a quincunx pattern. A quincunx is a geometric pattern that includes five dots arranged in a cross shape, with four dots forming a square or rectangle and a fifth dot in the center of the square or rectangle. In this example, the four pixels that form the square or rectangle are pixels of a first subframe and the fifth center pixel is a pixel of a second subframe. Another way to describe the pixels selected for the subframes is a checkerboard pattern. The subframes are described in more detail below in Figure 2

[0023] ​A first subframe of 1280x720 pixels is created by selecting a first subset of pixels that includes every other pixel in each row and every other pixel in each column of the first 2560x1440 rendered frame. A second subframe of 1280x720 pixels is created by shifting down one row and across one column and then again selecting a second subset of pixels that includes every other pixel in each row and every other pixel in each column of the first 2560x1440 rendered frame. The first subframe contains 1 / 4 of the pixels in the first 2560x1440 frame, and the second subframe also contains 1 / 4 of the pixels in the first 2560x1440 frame. The other 1 / 2 of the pixels in the first 2560x1440 frame are not used for the subframes, but are used for anti-aliasing purposes as described below. If four subframes are created instead of two subframes in another example, each of the four subframes contains 1 / 4 of the pixels in the 2560x1440 frame, and each pixel in the first 2560x1440 frame is used across the four subframes.

[0024] Figure 2 is a diagram of pixels 200 of a rendered frame in various examples. The pixels 200 are samples of pixels of a 2560x1440 frame of a rendered image for display. Sixteen of the 2560x1440 pixels are shown as Figure 2 pixels 202, 204, 206, and 208 are pixels of subframe A. Subframe A contains a first subset of 1280x720 pixels, but for simplicity only four pixels are shown here. The pixels of subframe A occupy every other row and every other column in a "tessellation" or "checkerboard" arrangement, as shown. In the tessellation, the pixels 202, 204, 206, and 208 form a square, and the center fifth pixel is pixel 212, which is a pixel of subframe B. Pixel 202 is shown in the first row and first column of this sample of pixels. Pixel 204 is in the first row and third column. Pixel 206 is in the third row and first column. Pixel 208 is in the third row and third column. The pixels in subframe A comprise a first subset of pixels selected from the full 2560x1440 image. To produce an image for display on the display 112 in the head-mounted viewer 100, the GPU 104 processes the 1280x720 pixels of subframe A to produce an image for display. Because subframe A is the first frame, subframe A is referred to as an "odd" frame. The GPU 104 will alternate displaying odd frames and even frames to produce an OSR image.

[0025] Pixels 212, 214, 216, and 218 are pixels of subframe B. Subframe B also contains a second subset of 1280x720 pixels, just like subframe A, but for simplicity only four pixels are shown here. In the quincunx arrangement, pixels 212, 214, 216, and 218 form a square, and the fifth pixel in the center is pixel 208, which is a pixel of subframe A. The pixels of subframe B occupy every other row and every other column in the checkerboard or quincunx arrangement, as shown. The pixels of subframe B are shifted down one row and to the right one column from the pixels of subframe A. For example, for each pixel in subframe B, pixel 212 is shifted down one row and to the right one column from pixel 202, and so on. The pixels within subframe B comprise a second subset of pixels selected from the full 2560x1440 image. Because subframe B is the second frame, subframe B is referred to as an “even” frame.

[0026] Pixels 220, 222, 224, 226, 228, 230, 232, and 234 are unsampled pixels of the 2560x1440 frame that are not used in subframe A or subframe B. Even though these pixels are not sampled for the subframes that are displayed, these unsampled pixels are used for anti-aliasing as described below. If four subframes are used instead of two, these unsampled pixels would be part of the third or fourth subframe in another example described below.

[0027] In operation, GPU 104 and other suitable components first process the pixels of subframe A. The pixels of subframe A are sampled from the first frame of 2560x1440 rendering (pixels 202, 204, 206, 208, etc.). In a process described below, an anti-aliasing filter is applied to the subframe A pixels to remove aliasing artifacts caused by the removal of the unsampled pixels (220, 222, 224, etc.). After filtering, GPU 104 transfers the 1280x720 pixels of subframe A to 1280x720 SLM 110 and associated circuitry and optics 114. SLM 110 receives the pixels of subframe A and projects the subframe A pixel information for display.

[0028] GPU 104 and other suitable components then process the pixels of subframe B. Note that due to multithreading, hyperthreading, and other techniques, the processing of the pixels of subframe A and subframe B can occur simultaneously, or even out of order. For subframe B, the pixels of subframe B are sampled from the first frame of 2560x1440 rendering (pixels 212, 214, 216, 218, etc.). An anti-aliasing filter is applied to the pixels of subframe B, and then GPU 104 transfers the 1280x720 pixels in subframe B to 1280x720 SLM 110 and associated circuitry and optics 114. SLM 110 receives the pixels of subframe B and projects the pixel information in subframe B for display. Compared to the subframe A pixels, the subframe B pixels will be optically shifted down one row and right one column, as the pixels of subframe B are positioned relative to the pixels of subframe A in FIG. 2B. Figure 2

[0029] After processing and displaying subframe B, the process again continues from subframe A, with sampling, filtering, and displaying from the second frame of 2560x1440 rendering. Subframe B is then sampled, filtered, and displayed from the second frame of 2560x1440 rendering. Subframes A and B continue to display in an alternating fashion as samples are taken from each subsequent frame of 2560x1440 rendering.

[0030] As described above, several spatially offset frames combined fast enough are perceived by a human observer to provide increased system resolution. The spatial positions cycle through in a relatively fast integration period, so the observer does not perceive the individual subframes, which would produce a flicker artifact. If all spatial positions are completed at 60 Hz or faster, the flicker artifact is eliminated. Thus, all spatial positions can be completed in 16.67 milliseconds or faster. The rate at which an observer perceives flicker is called the critical flicker fusion threshold. The critical flicker fusion threshold is proportional to the field of view. In some VR or AR headsets, display 112 produces a large image by filling a large angular range of the observer’s field of view. In one example, the field of view can be as wide as 40 degrees or more. For a wide field of view, a faster integration period is needed to prevent observed flicker. In the case of a 40-degree or more field of view, an integration period of 75 Hz or 90 Hz can provide better results than 60 Hz. In the example of FIG. 2, the two spatial positions are diagonally shifted by 1 / 2 pixel. The two spatial positions are rendered and displayed in a period faster than the critical flicker fusion period (e.g., 75 Hz). If the subframes are generated at 150 Hz, the integration of the two subframes is at half that rate or 75 Hz. Generating the two subframes at 150 Hz provides a 75 Hz integration period for each of the two subframes, which is fast enough to prevent or reduce perceived flicker. Figure 2

[0031] Figure 3 ​​is a diamond passband anti-aliasing filter according to an example. Graph 300 is a visual representation of a diamond passband filter. Equation 350 shows one example of the coefficients of a 3x3 finite impulse response (FIR) filter. In the above example, 2560x1440 samples were rendered, which is four times the number of micro-mirrors in the SLM 110 array. For sub-frame A, ¼ of the pixels were selected (pixels 202, 204, 206, 208, etc.). For sub-frame B, ¼ of the pixels were selected (pixels 212, 214, 216, 218, etc.). The other remaining pixels (pixels 220, 222, 224, 226, etc.) were not sampled. Because the remaining pixels were not sampled, the frequencies associated with these pixels were not represented in the samples of sub-frames A and B. Because these frequencies were not represented, the diamond passband filter was used on each sampled pixel of sub-frames B and A to cancel the frequencies and prevent or reduce aliasing artifacts.

[0032] Equation 350 is one example of the diamond passband filter coefficients. The diamond passband filter is used as a weighted sum. For example, for a 3x3 group of pixels, for all 9 pixels, the first pixel is multiplied by -1 / 16, the second pixel is multiplied by 2 / 16, the third pixel is multiplied by -1 / 16, the center pixel is multiplied by 12 / 16, etc. These nine products are added together to produce a single weighted sum, which replaces the pixel at the center of the 3x3 grid. The filter cancels the frequencies associated with the unsampled pixels, which are the diagonal frequencies in the frequency space of the center pixel of the 3x3 grid.

[0033] As one example of the diamond passband filter operation, pixel 212 of sub-frame B was selected to be filtered by the diamond passband filter. Pixel 212 is the center pixel of the 3x3 grid. The weighted sum of pixel 212 and the pixels surrounding pixel 212 is calculated using the coefficients in the 3x3 grid in equation 350. The nine pixels are weighted as follows: pixel 202 by -1 / 16, pixel 220 by 2 / 16, pixel 204 by -1 / 16, pixel 224 by 2 / 16, pixel 212 by 12 / 16, pixel 226 by 2 / 16, pixel 206 by -1 / 16, pixel 228 by 2 / 16, pixel 208 by -1 / 16. This weighted sum then replaces pixel 212, the center pixel in the 3x3 grid. The same filter is applied to each pixel in sub-frame B before the samples of sub-frame B are sent to the SLM 110 for display. This filtering process cancels the aliasing artifacts caused by the missing pixels (e.g., pixels 220, 224, 226, and 228).

[0034] Each pixel in subframe A is filtered before subframe A is passed to SLM 110, and each pixel in subframe B is filtered before subframe B is passed to SLM 110. Other filter coefficients can be used in other examples. In one example, GPU 104 performs the filtering operations to take advantage of the multi-threading capabilities of GPU 104.

[0035] Figure 4 An example filter span 400 across super-samples is shown in accordance with an example. Filter span 400 shows the diamond passband filter for subframe A pixels. Pixels 202, 204, 206, and 208 are shown as each being the center pixel of a 3x3 grid. The diamond passband filter described above with respect to Figure 3 The diamond passband filter described above with respect to

[0036] Figure 5 An example filter span 500 across super-samples is shown in accordance with an example. Filter span 500 shows the diamond passband filter for subframe B pixels. Pixels 212, 214, 216, and 218 are shown as each being the center pixel of a 3x3 grid. The diamond passband filter described above with respect to Figure 3The diamond passband filter is applied to every pixel 212, 214, 216, 218 and every other pixel in subframe B. For example, applying the diamond passband filter to pixel 212 includes weighting pixels 202, 220, 204, 224, 212, 226, 206, 228, and 208 according to the filter coefficients in equation 350. Applying the diamond passband filter to pixel 214 includes weighting pixels 204, 222, 502, 226, 214, 504, 208, 230, and 506. Applying the diamond passband filter to pixel 216 includes weighting pixels 206, 228, 208, 232, 216, 234, 510, 512, and 514. Applying the diamond passband filter to pixel 218 includes weighting pixels 208, 230, 506, 234, 218, 508, 514, 516, and 518. Similar filtering is performed for each of the 1280x720 pixels in subframe B. Figure 4 and Figure 5 A diagram 600 showing how the diamond passband filter is applied differently to odd frames (subframe A) and even frames (subframe B). For odd frames, the filter starts at pixel 202. For even frames, the filter starts at pixel 212, which is diagonally offset from pixel 202 by 1 / 2 pixel pitch.

[0037] Figure 6 is a diagram 600 of optical super resolution results according to an example. Diagram 600 shows results using a two-position super resolution architecture as described above, with the downstream SLM 110 optically shifted diagonally by 1 / 2 pixel pitch. The resolution of the display is shown in the lower left corner of diagram 600 and is labeled 602. In one example, the resolution of the display is 1280x720. Using OSR and a diagonal optical shift of 1 / 2 pixel, triangles 604 and 606 can be added. The vertical resolution and the horizontal resolution of the system are doubled. Finally, if a four- position super resolution architecture is used, then triangle 608 can be added to the resolution capabilities of the system, as described below. In one example, the four-position architecture would result in a resolution of 2560x1440 pixels.

[0038] Figure 7 is a diagram showing pixels 700 of a rendered frame in various examples. Pixels 700 are a sample of pixels of a 2560x1440 frame containing a rendered image for display. 16 of the 2560x1440 pixels are shown as Figure 7 . Figure 7 An example of a four-position super resolution architecture using four subframes instead of two subframes in the examples above is shown. Because four subframes are used, every pixel in the 2560x1440 frame is used, instead of half the pixels as described above in relation to Figure 2 .

[0039] In the four-position super-resolution architecture, pixels 202, 204, 206, and 208 are a first subset of pixels of a first sub-frame A. Pixels 212, 214, 216, and 218 are a second subset of pixels of a second sub-frame B. Pixels 220, 222, 228, and 230 are a third subset of pixels of a third sub-frame C. Pixels 224, 226, 232, and 234 are a fourth subset of pixels of a fourth sub-frame D. Each sub-frame includes one quarter of the pixels of the full-resolution frame.

[0040] As with the example above regarding Figure 2 , Figure 7 the sub-frames in are composed of alternating pixels in a checkerboard pattern. In this example, four sub-frames are sequentially sent to the SLM 110, each with a different spatial offset. The four sub-frames are sent to the SLM 110 and displayed with optical shifts at a fast enough rate to prevent the observer from perceiving flicker. Because four sub-frames are used, the perceived resolution of the display is four times the actual resolution of the display. Using the optical super-resolution described herein, a display with dimensions 1280x720 would appear to the user to be a display with a resolution of 2560x1440. The display would have a full resolution as shown in Figure 6 .

[0041] Figure 7 The sub-frames in the four-position super-resolution architecture of are delivered to the SLM 110 in any suitable order. The GPU 104 delivers the sub-frames to the SLM 110 after they are rendered. The SLM 110 receives the sub-frames and modulates the pixels of the SLM 110 to display on the display 112. In one example, sub-frame A is displayed first, then shifted one pixel to the right to display sub-frame C. Then, for sub-frame B, the display is shifted one pixel down. Finally, for sub-frame D, the display is shifted one pixel to the left. The four shifts occur at a fast enough rate to prevent perceived flicker. After sub-frame D is displayed, sub-frame A is displayed again, and the process of displaying the sub-frames is repeated in the order of A, C, B, and D.

[0042] When using the four-position resolution architecture, each pixel of the 2560x1440 image is used in one of the sub-frames. Because each pixel is used, the diamond passband filter described in Figure 3 above is not needed. After the sub-frames are created, they can be sent to the SLM 110 for display without needing to filter the pixels using the diamond passband filter. The filter is used to remove aliasing artifacts that result from the lack of frequencies associated with the unsampled pixels. Because there are no unsampled pixels in this example, the diamond passband filter is not needed.

[0043] Figure 8is an example screenshot 800 of a left side existing system and a right side optical super resolution system. Using the existing system, the pixelation along the edges of the rendered objects is apparent. On the right, the optical super resolution described herein reduces the pixelation along the same edges.

[0044] Figure 9 is an example screenshot 900 of a left side existing system and a right side optical super resolution system. Using the existing system, the horizontal lines separated by a single pixel in the screenshot are indistinguishable due to the low resolution. In contrast, the OSR system on the right creates a higher perceived resolution and the horizontal lines can be distinguished from one another even though the lines are separated by a single pixel.

[0045] Additional techniques are used in other various examples. One technique is spatially adaptive anti-aliasing. Spatially adaptive anti-aliasing is a technique that applies anti-aliasing only to high frequency edges of an image instead of the entire image. This technique reduces the computational load and reduces the power consumption of the GPU. In Figure 8 In the example of, a large portion of the image is simply blank. These areas do not require high resolution samples because there is little to no display information there. Anti-aliasing can be applied to areas of fine detail or edges, but not to solid color areas. The example here is produced in a closed ecosystem as described above, so the solid color areas within the render are known, as are the areas of fine detail and edges. This information is used to request additional samples and apply the diamond pass filter only to the areas of the image that require fine detail.

[0046] Another technique that can be employed in some examples is foveated rendering. Foveated rendering is a rendering technique that uses an eye tracker integrated with the head mounted viewer 100. The user perceives the highest resolution at the center of the user’s gaze. Far from the center of gaze, the resolution perceived by the user is much lower. In some cases, the center of gaze can be only ±3 degrees. Foveated rendering tracks the location of the user’s gaze and renders the image within the user’s gaze at high resolution while rendering the image outside the user’s gaze at low resolution. The computational load can be reduced by producing super samples and performing anti-aliasing filtering only within the user’s gaze. Reducing the computational load and power consumption provides benefits for AR head mounted viewers, such as extending battery life.

[0047] In another example, instead of generating a full resolution image (e.g., 2560x1440) and generating a subframe that reduces the resolution by a factor of four, two alternating frames can be rendered that are offset by a given spatial offset. By using two subframes in the example above, half of the pixels are used for anti-aliasing filtering, but do not directly correspond to the final output on the display because they are not part of subframe A or subframe B (e.g., Figure 2pixels 220, 222, 224, 226, etc.) In this example, instead of down-filtering the full resolution image, sub-frame A and sub-frame B are calculated directly in GPU 104. This method is referred to as a virtual lens shift method. The virtual camera is shifted in the rendering engine. The virtual camera is shifted so that the outputted pixels are the programmed pixels of the particular sub-frame. Instead of sampling the full resolution image, two 1280x720 frames are sent to the display. With this method, the anti-aliasing and other techniques performed by the rendering engine itself can be applied directly to the two sub-frames.

[0048] Figure 10 is a flowchart of an example method of optical super-resolution in various examples. Although the method steps are described in conjunction with Figures 1-9 any system configured to perform the method steps in any suitable order falls within the scope of this description.

[0049] Method 1000 begins at step 1010, where a GPU (e.g., GPU 104) renders a first sub-frame of an image, where the first sub-frame includes a first subset of pixels of the image. As described above, the first subset of pixels can include ¼ pixels of a high resolution image, and the first subset of pixels can include every other pixel in odd rows of pixels of the high resolution image.

[0050] Method 1000 continues at step 1020, where the first sub-frame is displayed on a display. In one example, the resolution of the first sub-frame matches the resolution of the display. In some examples, an SLM is used to project the sub-frame onto a display of a VR or AR headset.

[0051] Method 1000 continues at step 1030, where GPU 104 renders a second sub-frame of the image, where the second sub-frame includes a second subset of pixels of the image, and where the second sub-frame is diagonally shifted by half a pixel from the first sub-frame. As described above, the second subset of pixels can include ¼ pixels of a high resolution image, and the second subset of pixels can include every other pixel in even rows of pixels of the high resolution image. As shown and described above, sub-frame B is diagonally shifted by half a pixel from sub-frame A. Figure 2

[0052] Method 1000 continues at step 1040, where the second sub-frame is displayed on the display after the first sub-frame is displayed. The display is diagonally optically shifted by half a pixel to display the second sub-frame. As described above, in some examples, the first sub-frame and the second sub-frame are displayed at a rate of 75 Hz or faster to prevent or reduce flicker perceived by a user.

[0053] Figure 11 ​is an example method 1100 of optical super-resolution in various examples. The method 1100 is a method for determining when to use techniques such as foveation and anti-aliasing in conjunction with higher resolution supersamples, or when to instead produce lower resolution samples when lower resolution samples are sufficient to provide adequate optical quality. Although the method 1100 is described in the context of a GPU 104, any system configured to perform the method steps in any suitable order falls within the scope of this description. Figures 1-9 Method steps are described, but any system configured to perform the method steps in any suitable order falls within the scope of this description.

[0054] The method 1100 begins at decision block 1110, where a processing unit (such as the GPU 104) determines whether one or more pixels in an image are in a foveal region, and whether foveation is enabled. Gaze point rendering tracks the location of a user's gaze and renders images at high resolution within the user's gaze, while rendering images at low resolution outside the user's gaze. A sensor such as the sensor 108 is used to track the user's gaze. If foveation is enabled and one or more pixels are in the foveal region, the method proceeds to step 1150, where a higher resolution image is rendered. If foveation is not enabled or if one or more pixels are not in the foveal region, the method proceeds to decision block 1120.

[0055] At decision block 1120, the processing unit determines whether one or more pixels are in an edge region, and whether adaptive aliasing is enabled. If either answer is negative, the method proceeds to step 1130, where a lower resolution image is rendered. If one or more pixels are in an edge region of the image and adaptive aliasing is enabled, the method proceeds to step 1150, where a higher resolution image is rendered.

[0056] If the answers at decision blocks 1110 and 1120 are both negative, the method 1100 proceeds to step 1130. At step 1130, the processing unit produces a downsample of the pixels with appropriate spatial offset. An example of a spatial offset is shown in Figure 4 and Figure 5 where the sub-frames are diagonally offset by half a pixel. After step 1130, the method 1100 proceeds to step 1140.

[0057] At step 1140, appropriate output is routed to a display (such as the display 112). In some examples, an SLM (such as the SLM 110) is used to project the image onto the display. The SLM 110 receives pixel information for a frame and projects the pixel data for the frame for display.

[0058] If the answer is yes at decision block 1110 or decision block 1120, the method 1100 proceeds to step 1150. At step 1150, supersamples are generated from the rendered image having a higher resolution than the display resolution. One example of a higher resolution image is the 2560x1440 image described above. These supersamples provide a higher resolution image in the foveal region or the peripheral region to improve optical quality. After step 1150, the method 1100 proceeds to step 1160.

[0059] At step 1160, the processing unit applies an anti-aliasing filter to the supersamples with the appropriate spatial offset. The application of the anti-aliasing filter is described above with reference to Figures 3-5 At step 1160, the processing unit applies an anti-aliasing filter to the supersamples with the appropriate spatial offset. The application of the anti-aliasing filter is described above with reference to Figures 4-5 The spatial offset is shown. The application of the anti-aliasing filter can remove artifacts that can occur due to the lack of pixels in the supersamples. After step 1160, the method 1100 proceeds to step 1170.

[0060] At step 1170, the processing unit performs downsampling. The downsampling of the supersamples creates subframes, as described above with reference to Figure 2 At step 1170, the processing unit performs downsampling. The downsampling of the supersamples creates subframes, as described above with reference to

[0061] At step 1140, the processing unit routes the appropriate output to the display 112. In one example, this step includes transferring pixel information to the SLM 110, where the SLM 110 projects the pixels for display on the display 112. The output includes appropriate pixel data that is used to display the image at a fast enough rate to prevent the user from feeling flicker. After step 1140, the method 1100 proceeds to step 1180.

[0062] At step 1180, the rendered image is displayed on the display 112. At step 1190, the optomechanical system 122 optically shifts the display according to the rendering offset. As described above with reference to Figure 1 At step 1180, the rendered image is displayed on the display 112. At step 1190, the optomechanical system 122 optically shifts the display according to the rendering offset. As described above with reference to

[0063] The term“coupled” is used throughout the specification. This term can encompass a connection, a communication, or a signal path between components consistent with the description. For example, if device A generates a signal to control device B to perform an action, in a first example, device A is coupled to device B, or in a second example, device A is coupled to device B through intermediate component C if component C does not substantially alter the functional relationship between device A and device B, such that device B is controlled by device A via the control signal generated by device A.

[0064] A device“configured to” perform a task or function can be configured (e.g., programmed and / or hardwired) at a manufacturing or construction facility, and / or at an end user’s site, after manufacturing and / or construction to perform the function and / or another additional or alternative function. The configuration can be through firmware and / or software programming of the device, through a construction and / or layout and interconnection of hardware components of the device, or a combination thereof.

[0065] Circuits or devices described herein as including certain components can instead be adapted to be coupled to those components to form the described circuitry or device.

[0066] Although certain components can be described herein as components of a particular technology, these components can be exchanged with components of other technologies. Circuits described herein can be reconfigurable to include the exchanged components to provide at least a partial similarity to the functionality available prior to the component exchange.

[0067] Unless otherwise indicated, “about,”“approximately,” or“substantially” before a value indicates + / - 10% of the stated value. Modifications can be made to the described examples, and other examples can be made, within the scope of the claims.

Claims

1. A method for an optically superresolved display for rendering, the method comprising: in response to a determination by at least one processor that a region of pixels of an input image is in a foveal region, generating super-samples for the region of pixels from the input image to generate a rendered image; generating, by the at least one processor, a first sub-frame of the rendered image, wherein the first sub-frame comprises a first subset of pixels of the rendered image; displaying the first sub-frame to generate a first display image; generating, by the at least one processor, a second sub-frame of the rendered image, wherein the second sub-frame comprises a second subset of pixels of the rendered image, and wherein the second sub-frame is diagonally shifted by half a pixel from the first sub-frame; and displaying the second sub-frame after displaying the first sub-frame to generate a second display image, wherein the second display image is diagonally optically shifted by half a pixel relative to the first display image.

2. The method of claim 1, comprising displaying the first sub-frame and the second sub-frame at a rate of at least 75 hertz.

3. The method of claim 1, wherein the first subset of pixels comprises one quarter pixels of the input image, and the second subset of pixels comprises one quarter pixels of the input image.

4. The method of claim 3, wherein the first subset of pixels comprises every other pixel in odd rows of pixels of the input image.

5. The method of claim 3, wherein the second subset of pixels comprises every other pixel in even rows of pixels of the input image.

6. The method of claim 1, further comprising: generating a third sub-frame of the input image, wherein the third sub-frame comprises a third subset of pixels of the input image, and wherein the third sub-frame is shifted upward by half a pixel from the second sub-frame; displaying the third sub-frame after displaying the second sub-frame to generate a third display image, wherein the third display image is optically shifted upward by half a pixel relative to the second display image; generating a fourth sub-frame of the input image, wherein the fourth sub-frame comprises a fourth subset of pixels of the input image, and wherein the fourth sub-frame is diagonally shifted by half a pixel from the third sub-frame; and displaying the fourth sub-frame after displaying the third sub-frame to generate a fourth display image, wherein the fourth display image is diagonally optically shifted by half a pixel relative to the third display image to display the fourth sub-frame.

7. The method of claim 1, comprising applying an anti-aliasing filter to the first subset of pixels and the second subset of pixels.

8. The method of claim 7, comprising applying spatially adaptive anti-aliasing to edge regions of the input image.

9. The method of claim 1, wherein the region of pixels is a first region of pixels, the method further comprising: in response to a determination that a second region of pixels of the input image is outside the foveal region, generating a down-sampling of the second region of pixels with a spatial offset.

10. An image processing system comprising: a display; and ​ a processing unit coupled to the display and configured to: generate a first sub-frame of an image, wherein the first sub-frame comprises a first subset of pixels of the image, the image having the first subset of pixels, a second subset of pixels, and a third subset of pixels; apply an anti-aliasing filter to the first subset of pixels using the third subset of pixels; transfer the first sub-frame to the display; generate a second sub-frame of the image, wherein the second sub-frame comprises the second subset of pixels of the image, and wherein the second sub-frame is diagonally shifted by half a pixel from the first sub-frame; apply the anti-aliasing filter to the second subset of pixels using the third subset of pixels; and transfer the second sub-frame to the display.

11. The image processing system of claim 10, wherein the first subset of pixels and the second subset of pixels are in a foveal region.

12. The image processing system of claim 10, wherein the anti-aliasing filter is a 3x3 finite impulse response filter (3x3 FIR filter).

13. The image processing system of claim 10, wherein the processing unit transfers the first sub-frame and the second sub-frame to a spatial light modulator (SLM) at a rate of at least 75 Hz.

14. The image processing system of claim 10, wherein the first subset of pixels comprises one quarter of a pixel of the image, and the second subset of pixels comprises one quarter of a pixel of the image.

15. The image processing system of claim 10, wherein the anti-aliasing filter is a diamond passband anti-aliasing filter.

16. An optical display system comprising: a display; a processing unit configured to: render an image; generate a first sub-frame of the image, wherein the first sub-frame comprises a first subset of pixels of the image, the image having the first subset of pixels, a second subset of pixels, and a third subset of pixels; apply an anti-aliasing filter to the first subset of pixels using the third subset of pixels; generate a second sub-frame of the image, wherein the second sub-frame comprises the second subset of pixels of the image, and wherein the second sub-frame is diagonally shifted by half a pixel from the first sub-frame; and apply the anti-aliasing filter to the second subset of pixels using the third subset of pixels; and a spatial light modulator (SLM) coupled to the processing unit and optically coupled to the display, the SLM configured to: receive the first sub-frame from the processing unit and project the first sub-frame to the display to generate a first projected image; and receive the second sub-frame from the processing unit and project the second sub-frame to the display to generate a second projected image, wherein the second projected image is diagonally optically shifted by half a pixel relative to the first projected image.

17. The optical display system of claim 16, wherein the processing unit applies the anti-aliasing filter to pixels within a foveal region in the first subset of pixels and the second subset of pixels. ​ 18. The optical display system of claim 16, wherein the SLM is configured to project the first sub-frame and the second sub-frame at a rate of at least 75 Hz.

19. The optical display system of claim 16, wherein the first subset of pixels comprises one quarter of the pixels of the image and the second subset of pixels comprises one quarter of the pixels of the image.

20. The optical display system of claim 16, wherein the first subset of pixels is arranged in a quincunx arrangement.

Images