Apparatus and method for foveated rendering

Abstract

The display driver comprises image processing circuitry and drive circuitry. The image processing circuitry is configured to receive a foveated image, a full-frame image, and coordinate data specifying a location of the foveated image in the full-frame image. The image processing circuitry is further configured to generate a resultant image based on the full-frame image independent of the foveated image in response to detecting a data error in the coordinate data. The drive circuitry is configured to drive a display panel based on the resultant image.

Description

[Technical field]

[0001] The disclosed technology generally relates to an apparatus and method for foveated rendering. [Background technology]

[0002] Foveated rendering is a technique for generating a "foveated image" that may contain a foveated region that corresponds to a human's central vision and a peripheral region that corresponds to a human's peripheral vision. To mimic human vision, foveated rendering is sometimes performed by merging or overlaying a high-detail (or resolution) foveated image onto a lower-detail full-frame image. Summary of the Invention [Problem to be solved by the invention]

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] In one or more embodiments, a display driver is provided. The display driver includes image processing circuitry and drive circuitry. The image processing circuitry is configured to receive a foveated image, a full frame image, and coordinate data specifying a location of the foveated image in the full frame image. The image processing circuitry is further configured, in response to detecting a data error in the coordinate data, to generate a resultant image based on the full frame image independent of the foveated image. The drive circuitry is configured to drive a display panel based on the resultant image.

[0005] In one or more embodiments, a display system is provided. The display system includes a display panel and a display driver. The display driver is configured to receive a foveated image, a full-frame image, and coordinate data specifying a location of the foveated image in the full-frame image. The display driver is further configured to generate a resultant image based on the full-frame image, independent of the foveated image, in response to detecting a data error in the coordinate data. The display driver is further configured to drive the display panel based on the resultant image.

[0006] In one or more embodiments, a method of driving a display panel is provided, the method including receiving a foveated image, a full-frame image, and coordinate data specifying a location of the foveated image in the full-frame image, the method further including generating a resultant image based on the full-frame image independent of the foveated image in response to detecting a data error in the coordinate data, the method further including driving the display panel based on the resultant image.

[0007] Other aspects of the embodiments will be apparent from the following description and the appended claims. [Brief description of the drawings]

[0008] So that the above-mentioned features of the present disclosure can be understood in detail, a more particular description of the present disclosure briefly summarized above may be made with reference to embodiments, some of which are illustrated in the accompanying drawings. However, since the present disclosure admits of other equally effective embodiments, it should be noted that the accompanying drawings merely illustrate exemplary embodiments of the present disclosure and therefore should not be considered as limiting the scope of the invention.

[0009] [Figure 1] FIG. 1 illustrates an exemplary configuration of a display system according to one or more embodiments.

[0010] [Figure 2A]FIG. 2A illustrates an example of a resulting image in which a foveated image is merged with a full-frame image at a location specified by coordinate data, in accordance with one or more embodiments.

[0011] [Figure 2B] FIG. 2B illustrates an example of resulting image corruption in foveated rendering.

[0012] [Figure 3A] FIG. 3A illustrates an example configuration of a display driver according to one or more embodiments.

[0013] [Figure 3B] FIG. 3B illustrates an example configuration of a display driver according to one or more embodiments.

[0014] [Figure 4A] FIG. 4A illustrates an example data format of image data sent from the controller to the display driver during one frame period, according to one or more embodiments.

[0015] [Figure 4B] FIG. 4B illustrates another example of a data format for image data sent from the controller to the display driver during a frame period, according to one or more embodiments.

[0016] [Diagram 5] FIG. 5 illustrates an example method for foveated rendering in accordance with one or more embodiments.

[0017] For ease of understanding, the same reference numbers are used, where possible, to designate identical elements common to the figures. It is anticipated that elements disclosed in one embodiment may be beneficially used in other embodiments without specific mention. Reference numbers may be subscripted to distinguish identical elements from one another. The drawings referred to herein should not be understood to be drawn to scale unless specifically noted. Also, for clarity of presentation and explanation, the drawings are often simplified, omitting details or components. The drawings and discussion serve to explain the principles discussed below, with like numbers indicating like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The following detailed description is merely exemplary in nature and is not intended to limit the disclosed technology or the application or uses of the disclosed technology. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding background, brief summary or the following detailed description. As used herein, the term "coupled" means directly connected or connected via one or more intervening components or circuits.

[0019] Foveated rendering is a technique for generating a "foveated image" that may include a foveal region corresponding to the central human vision and a peripheral region corresponding to the peripheral vision. Foveated image rendering is widely used in head-mounted display (HMD) systems (e.g., virtual reality (VR) systems) to improve the user experience with less hardware.

[0020] One approach to achieve foveated rendering is to merge or overlay a foveated image with a full-frame image. The foveated image may correspond to the user's central vision, and the full-frame image may correspond to the user's entire field of view. To mimic human vision, the foveated image may contain more visual information (or detail) per unit area than the full-frame image. The location of the foveated image in the full-frame image may be determined based on the user's gaze direction, which may be detected by eye tracking.

[0021] A display driver (e.g., a display driver integrated circuit (DDIC)) configured to drive a display panel in an HMD system may support foveated rendering. In such a case, the display driver may be configured to receive a foveated image and a full-frame image from a controller external to the display driver and merge the foveated image with the full-frame image.

[0022] In various implementations, the controller may generate coordinate data, the coordinate data specifying a location of the foveal image in the full-frame image. The external controller further transmits the coordinate data to the display driver. The coordinate data may be generated based on a user's gaze direction detected by eye tracking. The display driver may be configured to merge the foveal image with the full-frame image such that the foveal image is located in the full-frame image at a location specified by the coordinate data.

[0023] One problem may be the occurrence of data errors in the coordinate data. Possible causes of the data errors may include electromagnetic interference (EMI) or noise applied to the display driver or the communication link between the display driver and the external controller, soft errors in the display driver and / or the external controller, and malfunctions of the display driver and / or the external controller. Data errors in the coordinate data may cause serious corruption of the foveated image generated by foveated rendering. The present disclosure provides various techniques to avoid corrupted foveated images being displayed on a display panel.

[0024] In one or more embodiments, a display driver includes image processing circuitry and driving circuitry. The image processing circuitry is configured to receive a foveated image, a full-frame image, and coordinate data specifying a position of the foveated image in the full-frame image. The image processing circuitry is further configured to generate a resultant image based on the full-frame image independent of the foveated image in response to detecting a data error in the coordinate data. Generating the resultant image based on the full-frame image independent of the foveated image may include using the full-frame image as the resultant image, or generating the resultant image by performing image processing (e.g., magnification) on the full-frame image without reference to the foveated image. The driving circuitry is configured to drive a display panel based on the resultant image. In the following, various embodiments of the present disclosure are described in detail.

[0025] 1 illustrates an exemplary configuration of a display system 1000 in accordance with one or more embodiments. The display system 1000 may be configured as a head mounted display (HMD) system, such as a virtual reality (VR) system. In the illustrated embodiment, the display system 1000 includes a display driver 100, a display panel 200, and a controller 300. The display panel 200 may be a liquid crystal display (LCD) panel, an organic light emitting diode (OLED) display panel, or other suitable display panel.

[0026] In the illustrated embodiment, the controller 300 is configured to send a foveated image and a full frame image to the display driver 100. The foveated image may correspond to the central visual field of the user, and the full frame image may correspond to the entire visual field of the user. In one or more embodiments, the full frame image may have the same resolution as the foveated image, but encompass a larger display area. In other embodiments, the full frame image may have a different resolution than the resolution of the foveated image. The display driver 100 is configured to generate a resultant image based on the foveated image and the full frame image, and drive the display panel 200 based on the resultant image. The resultant image may be a foveated image in which the foveated image is superimposed on the full frame image. In some implementations, the display driver 100 may be configured to display the resultant image on the display panel 200. In other implementations, the display driver 100 may be further configured to perform image processing on the resultant image and display an image resulting from the image processing on the display panel 200.

[0027] In various embodiments, display driver 100 is configured to enlarge the full-frame image received from controller 300 and merge the foveated image with the enlarged full-frame image to generate a resultant image. In one implementation, the enlarged full-frame image may be N times larger than the original full-frame image received from controller 300, where N is a magnification factor greater than 1. The magnification factor may be a natural number greater than or equal to 2. In some implementations, the foveated image and the original full-frame image have the same number of pixels, and the enlarged full-frame image may be N times larger than the original full-frame image (and the foveated image). 2 The foveal image may have twice as many pixels as the central image, in which case the non-foveal portion of the resulting image contains less visual information per unit area than the central image, effectively simulating peripheral vision.

[0028] In one or more embodiments, the controller 300 is further configured to send coordinate data to the display driver 100, the coordinate data specifying a location of the foveal image in the full-frame image. In such an embodiment, the display driver 100 is configured to generate a resultant image such that the foveal image is located in the magnified full-frame image at a location specified by the coordinate data. The coordinate data may indicate coordinates of a particular pixel (e.g., the pixel in the upper left corner) of the foveal image in the full-frame image or the magnified full-frame image. FIG. 2A illustrates an example of a resultant image in which the foveal image has been merged at a location (or coordinates) specified by the coordinate data, in accordance with one or more embodiments.

[0029] One problem with the display system 1000 illustrated in FIG. 1 may be a data error in coordinate data. Possible causes of the data error may include electromagnetic interference (EMI) or noise applied to the display driver 100 and / or the communication link between the display driver 100 and the controller 300, soft errors in the display driver 100 and / or the controller 300, and malfunctions of the display driver 100 and / or the controller 300. The data error in coordinate data may cause serious corruption of the resulting image. FIG. 2B illustrates an example of the corruption of the resulting image. When a data error occurs in the coordinate data, the foveal image may be merged with the enlarged full-frame image at an incorrect position, causing inconsistencies in the resulting image.

[0030] To mitigate or avoid corruption of the resulting image displayed on the display panel 200, in one or more embodiments, the display driver 100 is configured to generate a resulting image based on the full-frame image, independent of the foveated image, in response to detecting a data error in the coordinate data. In various implementations, the display driver 100 may be configured to output the upscaled full-frame image as the resulting image, without merging the foveated image with the upscaled full-frame image, in response to detecting a data error in the coordinate data. The display driver 100 may further be configured to generate a foveated image by merging the foveated image with the upscaled full-frame image, and output the foveated image as the resulting image, in response to not detecting a data error in the coordinate data. In such implementations, the resulting image may be either the foveated image or the upscaled full-frame image. The following describes an example embodiment of the display driver 100 configured in this manner.

[0031] 3A illustrates an exemplary configuration of a display driver 100 according to one or more embodiments. In the illustrated embodiment, the display driver 100 includes interface circuitry 102, graphic random access memories (GRAMs) 104, 106, image processing circuitry 108, and driver circuitry 110.

[0032] The interface circuitry 102 is configured to receive the foveal image and the full frame image from the controller 300. The foveal image and the full frame image may be transmitted to the interface circuitry 102 in the form of image data that includes pixel data for each pixel. The interface circuitry 102 is further configured to receive coordinate data from the controller 300. The coordinate data specifies the location of the foveal image in the full frame image.

[0033] FIG. 4A illustrates an example of a data format of image data transmitted from the controller 300 to the display driver 100 during one frame period, according to one or more embodiments. In FIG. 4A, each row indicates data transmitted during one horizontal period. "VS" indicates a vertical synchronization packet indicating the start of a frame period, and "HS" indicates a horizontal synchronization packet indicating the start of a horizontal synchronization period. "VBP" indicates a vertical back porch, "VFP" indicates a vertical front porch, "HBP" indicates a horizontal back porch, and "HFP" indicates a horizontal front porch. "Foveated image #1" to "Foveated image #M" collectively indicate image data of the foveated image. Here, "Foveated image #i" indicates image data of the i-th horizontal line of the foveated image, where i is an integer from 1 to M. Furthermore, "Full frame images #1 to #M" collectively indicate image data of the full frame image. Here, "Full frame image #i" indicates image data of the i-th horizontal line of the full frame image.

[0034] In the illustrated embodiment, the coordinate data is embedded as pixel data in the image data for the full frame image. In other embodiments, the coordinate data may be embedded in other locations in the image data. In yet other embodiments, the coordinate data may be sent to the display driver 100 separately from the image data.

[0035] 3A , interface circuitry 102 is further configured to transfer the foveated image to GRAM 104 and the full frame image to GRAM 106. Interface circuitry 102 may also be configured to process the foveated image and / or the full frame image and transfer the processed foveated image and / or full frame image to GRAM 104 and / or 106.

[0036] GRAM 104 is configured to store the foveated image received from interface circuitry 102, and GRAM 106 is configured to store the full frame image received from interface circuitry 102. In other embodiments, as shown in FIG. 3B, GRAM 106 may be omitted and the full frame image may be provided from interface circuitry 102 to image processing circuitry 108.

[0037] The image processing circuitry 108 is configured to generate a resultant image based on the foveated image and the full-frame image. In the illustrated embodiment, the image processing circuitry 108 includes a coordinate data detector 112, a magnification circuitry 114, and a synchronization and merging circuitry 116.

[0038] The coordinate data detector 112 is configured to detect data errors in the coordinate data. In embodiments where the coordinate data is embedded as pixel data in the image data of the full frame image, the coordinate data detector 112 may be configured to extract the coordinate data from the image data of the full frame image and detect data errors in the extracted coordinate data. Data error detection in the coordinate data may be based on a cyclic redundancy check or other error detection technique. In one implementation, the coordinate data detector 112 is configured to assert a coordinate data error flag in response to detecting a data error in the coordinate data and to deassert the coordinate data error flag in response to not detecting a data error in the coordinate data.

[0039] The enlargement circuitry 114 is configured to enlarge the full frame image and provide the enlarged full frame image to the synchronization and merging circuitry 116. In one implementation, the enlarged full frame image may be N times larger than the full frame image received from the controller 300, where N is an enlargement factor greater than 1. The enlargement factor N may be a natural number greater than or equal to 2. In some implementations, the enlarged full frame image may be N times larger than the original full frame image. 2 It may contain twice as many pixels.

[0040] The synchronization and merging circuitry 116 is configured to generate a resultant image based on the foveal image and the full frame image. The generation of the resultant image occurs in response to detection of a data error in the coordinate data. In various embodiments, the synchronization and merging circuitry 116 may be configured in response to non-detection of a data error in the coordinate data (e.g., in response to deassertion of the coordinate data error flag), to generate the resultant image by merging the foveal image with the enlarged full frame image such that the foveal image is located in the enlarged full frame image as specified by the coordinate data.

[0041] The synchronization and merging circuitry 116 may further be configured to generate a resultant image based on the upscaled full frame image and independent of the foveal image in response to detecting a data error in the coordinate data (e.g., in response to assertion of a coordinate data error flag). In one implementation, the synchronization and merging circuitry 116 may be configured to output the upscaled full frame image as a resultant image in response to detecting a data error in the coordinate data, without merging the foveal image with the upscaled full frame image. By omitting to merge the foveal image with the upscaled full frame image in response to detecting a data error in the coordinate data, corruption of the resultant image displayed on the display panel 200 is effectively mitigated or avoided.

[0042] The synchronization and merging circuitry 116 may further be configured to achieve synchronization between acquisition of the foveated image and the full frame image from the GRAMs 104 and 106 based on the coordinate data. In one implementation, the synchronization and merging circuitry 116 may be configured to acquire the foveated image from the GRAM 104 at a timing based on the coordinate data while successively acquiring the full frame images from the GRAM 106. Acquiring the foveated image at a timing determined based on the coordinate data may enable the foveated image to be merged with the full frame image at a location specified by the coordinate data.

[0043] Image processing circuitry 108 may be further configured to perform one or more other image processes on the resulting image produced by synchronization and merging circuitry 116 and forward the image produced by the image processing to driver circuitry 110. Image processing performed by image processing circuitry 108 may include color adjustment, gamma conversion, overdriving, or other image processing.

[0044] The driver circuitry 110 is configured to drive the display panel 200 based on the resultant image generated by the image processing circuitry 108. In some embodiments, the driver circuitry 110 is configured to drive the display panel 200 to display the resultant image on the display panel 200.

[0045] 4B illustrates another example of a data format for image data sent from controller 300 to display driver 100 during one frame period, according to one or more embodiments. The coordinate data may include multiple data portions, with each data portion containing the same data specifying the location of the foveal image. In the illustrated embodiment, the coordinate data includes two data portions embedded separately in the image data for the full frame image.

[0046] In an embodiment in which the coordinate data includes multiple data portions each including the same data specifying a location of the foveal image, as illustrated in FIG. 4B, the image processing circuitry 108 may be configured to detect a data error based on a mismatch of the multiple data portions. In one implementation, the image processing circuitry 108 may be configured to detect a data error when one of the multiple data portions differs from another one of the multiple data portions. In one implementation, the coordinate data detector 112 may be configured to extract a data portion of the coordinate data from image data of the full frame image and detect a data error in the coordinate data based on a mismatch of the data portions. In an embodiment in which the coordinate data includes two data portions as illustrated in FIG. 4B, the coordinate data detector 112 may be configured to detect a data error when the two data portions differ from each other.

[0047] Method 500 of Figure 5 illustrates example steps for achieving foveated rendering, according to one or more embodiments. It should be noted that one or more of the steps illustrated in Figure 5 may be omitted, repeated, and / or performed in a different order than that illustrated in Figure 5. Additionally, it should be noted that two or more steps may be performed simultaneously.

[0048] The method includes, in step 502, receiving a foveated image, a full frame image, and coordinate data indicating a position of the foveated image in the full frame image. Then, in step 504, in response to detection of a data error in the coordinate data, generating a resultant image based on the full frame image independent of the foveated image. The method further includes, in step 506, driving a display panel based on the resultant image. The method further includes, in step 508, receiving a second foveated image, a second full frame image, and second coordinate data indicating a position of the second foveated image in the second full frame image. Then, in step 510, in response to non-detection of a data error in the second coordinate data, generating a second resultant image by merging the second foveated image with the second full frame image. The method further includes, in step 512, driving a display panel based on the second resultant image.

[0049] While a number of embodiments have been described, those skilled in the art, having the benefit of this disclosure, will appreciate that other embodiments may be devised that do not depart from the scope of the disclosure. Accordingly, the scope of the disclosure should be limited only by the scope of the appended claims.

Claims

1. An image processing circuit unit, receiving a foveated image, a full-frame image, and coordinate data specifying a position of the foveated image in the full-frame image; an image processing circuit configured to generate a resultant image based on the full-frame image independently of the foveated image in response to detecting a data error in the coordinate data; a driving circuit configured to drive a display panel based on the resultant image; Equipped with Display driver.

2. the image processing circuitry is further configured to enlarge the full-frame image; generating the resultant image based on the full-frame image independently of the foveated image includes outputting the enlarged full-frame image as the resultant image in response to detecting the data error in the coordinate data.

2. The display driver according to claim 1.

3. The image processing circuit unit further receiving a second foveated image, a second full-frame image, and second coordinate data specifying a position of the second foveated image in the second full-frame image; configured to generate a second resultant image by merging the second foveated image with the second full-frame image in response to no data error being detected in the second coordinate data; the drive circuitry is further configured to drive the display panel based on the second result image.

2. The display driver according to claim 1.

4. the coordinate data is embedded in the full-frame image as pixel data; the image processing circuitry is configured to extract the coordinate data from the full-frame image; 2. The display driver according to claim 1.

5. A display panel; A display driver, receiving a foveated image, a full-frame image, and coordinate data specifying a position of the foveated image in the full-frame image; generating a resultant image based on the full-frame image independently of the foveated image in response to detecting a data error in the coordinate data; a display driver configured to drive a display panel based on the resultant image; Equipped with Display system.

6. the display driver is further configured to enlarge the full-frame image; generating the resultant image based on the full frame image independently of the foveated image includes outputting the enlarged full frame image as the resultant image in response to the detection of the data error in the coordinate data. The display system of claim 5 .

7. The display driver further receiving a second foveated image, a second full-frame image, and second coordinate data specifying a position of the second foveated image in the second full-frame image; generating a second resultant image by merging the second foveated image with the second full-frame image in response to no data error being detected in the second coordinate data; configured to drive the display panel based on the second resultant image. The display system of claim 5 .

8. receiving a foveated image, a full-frame image, and coordinate data specifying a location of the foveated image in the full-frame image; generating a resultant image based on the full-frame image independently of the foveated image in response to detecting a data error in the coordinate data; driving a display panel based on the resulting image; Contains method.

9. further comprising enlarging the full-frame image; generating the resultant image based on the full-frame image independently of the foveated image includes outputting the enlarged full-frame image as the resultant image in response to detecting the data error in the coordinate data. The method of claim 8.

10. receiving a second foveated image, a second full-frame image, and second coordinate data specifying a location of the second foveated image in the second full-frame image; generating a second resultant image by merging the second foveated image with the second full-frame image in response to no data error being detected in the second coordinate data; driving the display panel based on the second result image; Further includes The method of claim 8.

Images