Device and method for foveated rendering

Abstract

To provide a display driver which reduces processing overhead of an HMD device.SOLUTION: A display driver includes interface circuitry, image processing circuitry, and drive circuitry. The interface circuitry is configured to receive a full frame image and a foveal image from a source external to the display driver. The image processing circuitry is configured to: upscale the full frame image and render a foveated image from the upscaled full frame image and the foveal image. The foveated image includes a foveal area based on the foveal image, a peripheral area based on the upscaled full frame image, and a border area based on the foveal image and the upscaled full frame image. The border area is located between the foveal area and the peripheral area. The drive circuitry is configured to drive a display panel using the foveated image.SELECTED DRAWING: Figure 7A

Description

Technical Field

[0001] The disclosed technology generally relates to apparatus and methods for foveated rendering.

Background Art

[0002] A head-mounted display (HMD) device is configured to be worn or otherwise attached to a user's head. The HMD device may include one or more displays positioned in front of one or both of the user's eyes. More specifically, the HMD device is configured to display an image or video in close proximity to the user's eyes to provide an immersive experience to the user of the HMD device. Thus, the HMD device is suitable for augmented reality (XR) applications (including, inter alia, virtual reality (VR), augmented reality (AR) and mixed reality (MR)). However, because the display is in close proximity to the user's eyes, only a relatively small portion of the displayed image reaches the fovea of the user. In other words, the user's eyes can only focus on a relatively small area of each display at any given time.

[0003] Foveated rendering is a display technology often associated with HMD devices. More specifically, foveated rendering can reduce the processing overhead of an HMD device by dynamically rendering different regions of an image at different resolutions (also referred to as a "foveated image"). For example, a foveated image may include a foveal region and a peripheral region. The foveal region is aligned with the user's fovea and thus represents the region of the image that can be perceived with the highest visual acuity. On the other hand, the peripheral region is the region surrounding the foveal region and thus is in the user's peripheral vision. Foveated rendering may simulate human vision by, for example, rendering the peripheral region at a lower resolution (or lower level of detail) than the foveal region. [Overview of the project]

[0004] This abstract is provided in a concise form to introduce the selection of concepts further described below in modes for carrying out the invention. This abstract is not intended to identify any major or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] In some embodiments, the display driver comprises an interface circuit, an image processing circuit, and a drive circuit. The interface circuit is configured to receive a full-frame image and a foveal image from an external source to the display driver. The image processing circuit is configured to upscale the full-frame image and render a foveated image from the upscaled full-frame image and the foveal image. The foveated image includes a foveal region based on the foveal image, a peripheral region based on the upscaled full-frame image, and a boundary region based on the foveal image and the upscaled full-frame image. The boundary region is located between the foveal region and the peripheral region. The drive circuit is configured to drive the display panel using the foveated image.

[0006] In some embodiments, the display system comprises a source, a display panel, and a display driver. The source is configured to render a full-frame image and a foveal image. The display driver is configured to receive the full-frame image and the foveal image from the source. The display driver is further configured to upscale the full-frame image and render a foveated image from the upscaled full-frame image and the foveal image. The foveated image includes a foveal region based on the foveal image, a peripheral region based on the upscaled full-frame image, and a boundary region based on the foveal image and the upscaled full-frame image. The boundary region is located between the foveal region and the peripheral region. The display driver is further configured to drive the display panel using the foveated image.

[0007] In some embodiments, a method is provided for operating a display driver. The method includes receiving a full-frame image and a foveal image from a source external to the display driver. The method further includes upscaling the full-frame image. The method further includes rendering a foveated image from the upscaled full-frame image and the foveal image. The foveated image includes a foveal region based on the foveal image, a peripheral region based on the upscaled full-frame image, and a boundary region based on the foveal image and the upscaled full-frame image. The boundary region is located between the foveal region and the peripheral region. The method further includes driving a display panel using the foveated image. [Brief explanation of the drawing]

[0008] This embodiment is illustrated as an example and is not intended to be limited to the form shown in the accompanying drawings.

[0009] [Figure 1] Figure 1 illustrates an exemplary configuration of a display system compatible with foveated rendering, according to one or more embodiments.

[0010] [Figure 2] Figure 2 illustrates exemplary operation of a display driver according to one or more embodiments.

[0011] [Figure 3] Figure 3 illustrates an exemplary foveated image with a non-smooth boundary in the foveal image.

[0012] [Figure 4A] Figure 4A illustrates an exemplary configuration of the display system.

[0013] [Figure 4B]FIG. 4B illustrates an exemplary architecture of a display system for efficient boundary smoothing according to one or more embodiments.

[0014] [Figure 4C] FIG. 4C illustrates an exemplary configuration of a resulting image according to one or more embodiments.

[0015] [Figure 5] FIG. 5 illustrates an exemplary configuration of a controller according to one or more embodiments.

[0016] [Figure 6] FIG. 6 illustrates an exemplary data format of image data transmitted from a controller to a display driver during one frame period according to one or more embodiments.

[0017] [Figure 7A] FIG. 7A illustrates an exemplary configuration of a display driver according to one or more embodiments.

[0018] [Figure 7B] FIG. 7B illustrates an exemplary integration of a foveated image and an upscaled full-frame image for rendering a foveated image according to one or more embodiments. ]>

[0019] [Figure 8] FIG. 8 illustrates an exemplary configuration of a foveated image according to one or more embodiments.

[0020] [Figure 9] FIG. 9 illustrates exemplary weights assigned to pixels of a foveated image when determining the gradation of pixels in a boundary region as a weighted sum according to one or more embodiments.

[0021] [Figure 10] FIG. 10 illustrates an exemplary foveated image according to one or more embodiments.

[0022] [Figure 11] Figure 11 illustrates exemplary weights assigned to pixels in a foveal image according to one or more embodiments.

[0023] [Figure 12] Figure 12 illustrates an exemplary configuration of a foveated image according to one or more embodiments.

[0024] [Figure 13] Figure 13 illustrates exemplary weights assigned to pixels in a foveal image when determining the grayscale of boundary region pixels as a weighted sum according to one or more embodiments.

[0025] [Figure 14] Figure 14 illustrates an exemplary configuration of a register according to one or more embodiments.

[0026] [Figure 15] Figure 15 is an exemplary flowchart illustrating an exemplary method of operating a display driver according to one or more embodiments.

[0027] For ease of understanding, where possible, the same reference numerals are used to indicate identical elements common to the drawings. Elements disclosed in one embodiment are expected to be usefully used in other embodiments, even without specific mention. Reference numerals may be subscripted to distinguish identical elements from one another. Drawings referenced herein should not be understood to be dimensional unless otherwise noted. Also, for clarity of presentation and explanation, drawings are often simplified, with details or components omitted. The drawings and discussions are intended to illustrate the principles discussed below, and similar numerals indicate similar elements. [Modes for carrying out the invention]

[0028] The detailed description below is essentially illustrative and is not intended to limit the Disclosure or its uses and applications. Furthermore, it is not intended to be bound by any explicit or implicit theories presented in the preceding background, summary, or the detailed description below. In this specification, the term “combined” means directly connected or connected via one or more intervening components or circuits.

[0029] Foveated rendering is a display technique often used to reduce the processing overhead of HMD devices by dynamically rendering different areas of an image at different resolutions (also known as "foveated images"). More specifically, foveated rendering can mimic human vision by rendering the peripheral areas of an image at a lower resolution (or lower fineness) than the foveal area of ​​the image. For example, the foveal area may be aligned with the user's fovea, while the peripheral area may be in the user's peripheral field of vision.

[0030] Foveated images can be generated by integrating or overlaying a foveal image with a full-frame image. The foveal image may be displayed or rendered in the foveal region of the foveated image, and the full-frame image may be displayed or rendered in the peripheral region of the foveated image. Similar to human vision, the foveal image may contain more information (or detail) per unit area than the full-frame image. The placement or position of the foveal image relative to the full-frame image may be determined based on the user's gaze direction. This gaze direction may be detected by eye tracking.

[0031] HMD systems may include a display driver (e.g., a display driver integrated circuit (DDIC)) configured to drive the display panel to display or render a foveated image. More specifically, the display driver may be configured to render a foveated image by integrating the foveal image with the full-frame image. In some implementations, the display driver may be configured to receive the foveal image and the full-frame image from a source via a communication link.

[0032] The communication link between the display device and the source may have limited bandwidth. To achieve high latency throughput with such limited bandwidth, the source may send a full-frame image to the display driver at its intrinsic resolution (which may be lower than the display's resolution). The display driver may then upscale that full-frame image and integrate the foveal image with the upscaled full-frame image for display.

[0033] Differences in resolution between full-frame images and foveal images can result in "unsmooth" boundaries at the edges of the foveal region in foveated images. More specifically, the transition from the foveal region to the peripheral region may appear rigid or jarring due to the change in resolution. This disclosure provides an efficient method for smoothing the boundaries of foveal images in foveated images.

[0034] In one or more embodiments, the display driver comprises an interface circuit, an image processing circuit, and a drive circuit. The interface circuit is configured to receive a full-frame image and a foveal image from an external source to the display driver. The image processing circuit is configured to upscale the full-frame image, render the peripheral region of the foveated image based on the upscaled full-frame image, render the foveal region of the foveated image based on the foveal image, and render the boundary region of the foveated image based on the first part of the foveal image together with the second part of the upscaled full-frame image. The foveal region is framed within the peripheral region, and the boundary region is located between the foveal region and the peripheral region. The drive circuit is configured to drive the display panel based on the foveated image.

[0035] Figure 1 illustrates an exemplary configuration of a foveated rendering display system 1000 according to one or more embodiments. In the illustrated embodiment, the display system 1000 comprises a controller 100 and an HMD device 600. The HMD device 600 comprises a display driver 200 and a display panel 300. The display panel 300 may be a liquid crystal display (LCD) panel, an organic light-emitting diode (OLED) display panel, or a display panel that implements various other suitable display technologies. The HMD device 600 further comprises a lens 610 positioned in front of the display panel 300. The user of the HMD device 600 views the image displayed on the display panel 300 by looking through the lens 610.

[0036] The controller 100 is configured as a source to supply the display driver 200 with a foveal image and a full-frame image. The foveal image may correspond to the foveal region of the foveated image that is positionally aligned with the user's fovea (for example, the region of the image in which the user's eyes are in focus), and the full-frame image may correspond to the peripheral region of the foveated image that is in the user's peripheral field of view. In some embodiments, the full-frame image may have the same intrinsic resolution as the foveal image but encompass a wider display area than the foveal image. In other embodiments, the full-frame image may have a different intrinsic resolution than the foveal image. The display driver 200 is configured to render the foveated image based on the foveal image and the full-frame image, and to drive the display panel 300 based on the foveated image.

[0037] Figure 2 illustrates exemplary operation of the display driver 200 according to one or more embodiments. In some embodiments, the display driver 200 is configured to receive a full-frame image 201 and a foveal image 202 from the controller 100. The display driver 200 is further configured to upscale the full-frame image 201 received from the controller 100 and integrate the foveal image 202 with the upscaled full-frame image indicated by reference numeral 203 to generate a foveated image 204. In the foveated image 204, the foveal image 202 is superimposed on the upscaled full-frame image 203. The display driver 200 is further configured to drive or render the foveated image 204 on the display panel 300. In some implementations, the display driver 200 may display the foveated image 204 directly on the display panel 300. In other implementations, the display driver 200 may perform further image processing on the foveated image 204 before displaying the image on the display panel 300. In some implementations, the full-frame image 201 may be upscaled by a magnification N, where N is an integer greater than 1. In some implementations, the foveal image 202 has the same number of pixels as the original full-frame image 201, and the upscaled full-frame image 203 has N of the original full-frame image 201 (and foveal image 202). 2 It may contain twice the number of pixels. Therefore, the upscaled full-frame image 203 contains less visual information per unit area than the foveal image.

[0038] As discussed above, integrating the foveal image 202 with the upscaled full-frame image 203 can result in a less smooth boundary around the foveal image 202. Figure 3 illustrates an exemplary foveated image 205 where the foveal image 206 is superimposed on the upscaled full-frame image 207. The foveated image 205 shows a less smooth boundary around the foveal image 206. This less smooth boundary can be perceptible to the user and even irritating, potentially degrading the user experience.

[0039] Aspects of this disclosure may mitigate the rough edges of a foveal image by performing boundary smoothing on the foveal image in a controller and transmitting the boundary-smoothed foveal image to a display driver. Figure 4A illustrates an exemplary configuration of a display system 2000 configured for boundary smoothing. The illustrated display system 2000 comprises a controller 700, a display driver 800, and a display panel 900. The controller 700 is configured as a source configured to render a foveal image and a full-frame image. The controller 700 is further configured to perform boundary smoothing on the foveal image and transmit the boundary-smoothed foveal image and the full-frame image to the display driver 800. The display driver 800 is configured to upscale the full-frame image and integrate the boundary-smoothed image with the upscaled full-frame image to render a foveated image displayed on the display panel 900.

[0040] Boundary smoothing based on the upscaled full-frame image (rather than the original full-frame image) can increase the processing overhead of the controller 700. Furthermore, in situations where the controller 700 is not fully aware of the specifications of the display driver 800 (for example, if the controller 700 and the display driver 800 are manufactured by different vendors), upscaling may need to be performed by the display driver 800.

[0041] Figure 4B illustrates an exemplary architecture of a display system 1000A for efficiently achieving boundary smoothing according to one or more embodiments. In the illustrated embodiment, a controller 100A is configured as a source that provides a foveal image and a full-frame image to a display driver 200A. The display driver 200A is configured to upscale the full-frame image and integrate the foveal image with the upscaled full-frame image to generate a foveated image to be displayed on the display panel 300. The display driver 200A is further configured to perform boundary smoothing when integrating the foveal image with the upscaled full-frame image.

[0042] Figure 4C illustrates an exemplary configuration of a foveated image 400 according to one or more embodiments. In one or more embodiments, the foveated image 400 includes a peripheral region 410, a foveal region 420, and a boundary region 430. The foveal region 420 is framed by or surrounded by the peripheral region 410, and the boundary region 430 is located between the peripheral region 410 and the foveal region 420. The display driver 200A is configured to render the peripheral region 410 of the foveated image 400 based on the upscaled full-frame image and to render the foveal region 420 of the foveated image 400 based on the foveal image. The display driver 200A is further configured to render the boundary region 430 by blending the pixels of the foveal image with the pixels of the upscaled full-frame image within the boundary region 430 of the foveated image 400 to perform boundary smoothing. Please note that the shapes of the peripheral region 410, the foveal region 420, and the boundary region 430 may differ from those shown in the illustration. The shapes of the peripheral region 410, the foveal region 420, and the boundary region 430 may be regular (as illustrated in Figure 4C, for example) or irregular.

[0043] Figure 5 illustrates an exemplary configuration of the controller 100A of Figure 4B according to one or more embodiments. In the illustrated embodiments, the controller 100A comprises a rendering engine 110 and an interface circuit 120. In some embodiments, the rendering engine 110 is configured to render a foveal image and a full-frame image. In other embodiments, the controller 100A may be configured to receive a foveal image and a full-frame image from an entity outside of the controller 100A. The rendering engine 110 may further be configured to generate control data used to control the integration of the foveal image with the full-frame image in the display driver 200A, including the rendering of the boundary region 430 (illustrated in Figure 4C). In some implementations, the control data may include coordinate data indicating the position (to be superimposed on the upscaled full-frame image) relative to the upscaled full-frame image. The coordinate data may indicate the coordinates of the foveal image in the foveated image 400. The control data may further include boundary region control data specifying one or more dimensions of the boundary region 430. For example, boundary region control data may specify the size of the boundary region 430. In some embodiments, the control data may be embedded as pixel data of the full-frame image. For example, the pixel data may be embedded in a portion of the full-frame image that is not displayed on the display panel 300 (e.g., the corners of the display area of ​​the display panel 300). The interface circuit 120 is configured to transmit the foveal image and the full-frame image as image data to the display driver 200A. The image data may include the gradation of pixels in the foveal image and the full-frame image. In some embodiments, the interface circuit 120 may be further configured to transmit the control data to the display driver 200A separately from the image data of the full-frame image.

[0044] Figure 6 illustrates an exemplary data format of image data transmitted from controller 100A to display driver 200A during one frame period according to one or more embodiments. In Figure 6, each row represents data transmitted during one horizontal synchronization period. "VS" indicates a vertical synchronization packet indicating the start of a frame period, and "HS" indicates a horizontal synchronization packet indicating the start of a horizontal synchronization period. "VBP" indicates a vertical back porch, "VFP" indicates a vertical front porch, "HBP" indicates a horizontal back porch, and "HFP" indicates a horizontal front porch. "Foveous Image #1" to "Foveous Image #M" collectively represent image data of the foveal image. Here, "Foveous Image #i" represents image data of the i-th horizontal line of the foveal image, where i is an integer from 1 to M. Furthermore, "Full Frame Image #1" to "Full Frame Image #M" collectively represent image data of the full frame image. Here, "Full Frame Image #i" represents image data of the i-th horizontal line of the full frame image. In the illustrated embodiment, the control data is embedded as pixel data in the full-frame image. In other embodiments, the coordinate data may be embedded at different locations in the full-frame image.

[0045] Figure 7A illustrates an exemplary configuration of a display driver 200A according to one or more embodiments. In the illustrated embodiments, the display driver 200A comprises an interface circuit 210, graphics random access memory (GRAM) 220, 230, an image processing circuit 240, a register 250, and a drive circuit 260. In some implementations, the display driver 200A may be configured as a display driver integrated circuit (DDIC) in which the interface circuit 210, GRAM 220, 230, image processing circuit 240, register 250, and drive circuit 260 are monolithically integrated on a single semiconductor chip.

[0046] The interface circuit unit 210 is configured to receive a foveal image and a full-frame image from the controller 100A. The interface circuit unit 210 is further configured to transfer the foveal image to the GRAM 220 and the full-frame image to the GRAM 230. The interface circuit unit 210 may also be configured to process the foveal image and / or the full-frame image and transfer the processed foveal image and / or full-frame image to the GRAM 220 and / or 230. In an embodiment in which control data is transmitted to the display driver 200A separately from the full-frame image, the interface circuit unit 210 may also be configured to receive control data from the controller 100A and transfer the control data to the image processing circuit unit 240. The GRAM 220 is configured to store the foveal image received from the interface circuit unit 210, and the GRAM 230 is configured to store the full-frame image received from the interface circuit unit 210. In other embodiments, the GRAM 230 may be omitted, and the full-frame image may be supplied from the interface circuit unit 210 to the image processing circuit unit 240. The image processing circuit 240 is configured to upscale the full-frame image and render a foveated image based on the foveal image and the upscaled full-frame image. The foveated image may also be rendered by integrating the foveal image with the upscaled full-frame image. The register 250 is configured to store boundary region dimension data indicating one or more dimensions of the boundary region 430 (illustrated in Figure 4C). The boundary region dimension data may define the configuration and / or shape of the boundary region 430. The drive circuit 260 is configured to drive the display panel 300 to display the foveated image on the display panel 300.

[0047] In the illustrated embodiment, the image processing circuit 240 includes a control data detector 242, an upscaling circuit 244, an integration circuit 246, an address counter 248, and a weight determination circuit 252. The control data detector 242 is configured to extract control data embedded in the full-frame image and transfer the control data to the integration circuit 246. In some embodiments, the control data detector 242 may further be configured to extract coordinate data (indicating the position of the foveal image relative to the upscaled full-frame image) from the image data of the full-frame image and to detect data errors in the extracted coordinate data. Data error detection of the coordinate data may be based on cyclic redundancy check or other error detection techniques. In some implementations, the control data detector 242 is configured to assert a coordinate data error flag when a data error is detected in the coordinate data and to deassert the coordinate data error flag when no data errors are detected in the coordinate data.

[0048] The upscaling circuit 244 is configured to upscale the full-frame image and provide the upscaled full-frame image to the integration circuit 246. In some implementations, the upscaled full-frame image may be N times larger than the full-frame image received from the controller 100A, where N is an integer greater than 1. In some implementations, the upscaled full-frame image is N times larger than the original full-frame image. 2 It contains twice the number of pixels.

[0049] The integration circuit 246 is configured to render a foveated image by integrating the foveal image with the upscaled full-frame image. As described in relation to Figure 4C, the foveated image 400 includes a peripheral region 410, a foveal region 420, and a boundary region 430. In one or more embodiments, the integration circuit 246 is configured to render the peripheral region 410 of the foveated image 400 based on the upscaled full-frame image and to render the foveal region 420 of the foveated image 400 based on the foveal image. The integration circuit 246 is further configured to render the boundary region 430 of the foveated image 400 by blending the pixels of the foveal image with the pixels of the upscaled full-frame image in the boundary region 430 to achieve boundary smoothing. In one or more embodiments, the pixels of the foveal image may be blended with the pixels of the upscaled full-frame image using alpha blending technique.

[0050] In some embodiments, the integration circuit 246 may render the resulting image depending on whether a data error is detected in the coordinate data. In some embodiments, when no data error is detected in the coordinate data (for example, in response to deassertion of the coordinate data error flag), the integration circuit 246 may render the foveated image as the resulting image by integrating the foveal image with the upscaled full-frame image so that the foveal image is positioned in the upscaled full-frame image as specified by the coordinate data. In some other embodiments, when a data error is detected in the coordinate data (for example, in response to assertion of the coordinate data error flag), the integration circuit 246 may output the upscaled full-frame image as the resulting image without integrating the foveal image with the upscaled full-frame image. By not integrating, the collapse of the resulting image displayed on the display panel 300 can be effectively mitigated or avoided.

[0051] Figure 7B illustrates an exemplary integration of a foveal image (indicated by reference numeral 401) and an upscaled full-frame image (indicated by reference numeral 402) for rendering a foveated image 400, according to one or more embodiments. In the illustrated embodiments, the foveal image 401 has an inner portion 403 and an outer portion 404, and the upscaled full-frame image 402 has an inner portion 405, an outer portion 406 and an intermediate portion 407. The inner portion 403 and outer portion 404 of the foveal image 401 correspond to the foveal region 420 and boundary region 430 of the foveated image 400, respectively. More specifically, the shape and position of the inner portion 403 of the foveal image 401 coincide with the shape and position of the foveal region 420 of the foveated image 400, and the shape and position of the outer portion 404 of the foveal image 401 coincide with the shape and position of the boundary region 430 of the foveated image 400. The inner portion 405, outer portion 406, and intermediate portion 407 of the upscaled full-frame image 402 correspond to the foveal region 420, peripheral region 410, and boundary region 430 of the foveated image 400, respectively. More specifically, the shape and position of the inner portion 405 of the upscaled full-frame image 402 coincide with the shape and position of the foveal region 420 of the foveated image 400, and the shape and position of the outer portion 406 of the upscaled full-frame image 402 coincide with the shape and position of the peripheral region 410 of the foveated image 400. Furthermore, the shape and position of the intermediate portion 407 of the upscaled full-frame image 402 coincide with the shape and position of the boundary region 430 of the foveated image 400.

[0052] In some embodiments, the pixel values ​​of the foveated image 400's foveated region 420 are obtained directly (or exclusively) from the pixel values ​​of the inner portion 403 of the foveated image 401, and the pixel values ​​of the foveated image 400's peripheral region 410 are obtained directly (or exclusively) from the pixel values ​​of the outer portion 406 of the upscaled full-frame image 402. In some embodiments, the pixel values ​​of the foveated image 400's boundary region 430 are obtained from the pixel values ​​of the outer portion 404 of the foveated image 401 and from the pixel values ​​of the middle portion 407 of the upscaled full-frame image 402. More specifically, the pixel values ​​of the outer portion 404 of the foveated image 401 are integrated or blended with the pixel values ​​of the middle portion 407 of the upscaled full-frame image 402 to render the pixel values ​​of the foveated image 400's boundary region 430.

[0053] In some embodiments, the integration of the outer portion 404 of the foveal image 401 and the middle portion 407 of the upscaled full-frame image 402 may include determining the tonality of the target pixel 411 in the boundary region 430 based on a weighted sum of the tonality of the corresponding pixel 412 in the outer portion 404 of the foveal image 401 and the tonality of the corresponding pixel 413 in the middle portion 407 of the upscaled full-frame image 402. Here, the corresponding pixel 412 in the outer portion 404 of the foveal image 401 and the corresponding pixel 413 in the middle portion 407 of the upscaled full-frame image 402 correspond to the position of the target pixel 411 in the boundary region 430. In some implementations, the tonality of the target pixel 411 may be determined as a weighted sum of the tonality of the corresponding pixel 412 in the foveal image 401 and the tonality of the corresponding pixel 413 in the upscaled full-frame image 402. Tonality P of the target pixel 411 T This may be determined according to the following formula (1). (1) P T =w1P1+w2P2 Here, w1 is the weight assigned to the corresponding pixel 412 of the foveal image 401, P1 is the grayscale of the corresponding pixel 412, w2 is the weight assigned to the corresponding pixel 413 of the foveal image 401, and P2 is the grayscale of the corresponding pixel 413.

[0054] Returning to Figure 7A, the address counter 248, register 250, and weight determination circuit 252 are configured together to determine the weights to be assigned to the corresponding pixels 412 in the foveated image 401 and the corresponding pixels 413 in the upscaled full-frame image 402 in the weighted sum. The address counter 248 is configured to determine the position of the target pixel 411 in the foveated image 400 whose grayscale is to be determined. In one implementation, the address counter 248 may be configured to determine the position of the target pixel 411 by counting the dot clock signal, the horizontal synchronization signal, and the vertical synchronization signal. The register 250 is configured to store a boundary region dimension data set indicating one or more dimensions of the boundary region 430 (illustrated in Figure 4C). The boundary region dimension data may indicate the configuration and / or shape of the boundary region 430.

[0055] In some embodiments, the weight determination circuit 252 is configured to determine the weight to be assigned to the corresponding pixel 412 in the foveated image 401 and the weight to be assigned to the corresponding pixel 413 in the upscaled full-frame image 402, based on the position of the target pixel 411 in the foveated image 400, when determining the gradation of the target pixel 411 in the foveated image 400. The weight determination circuit 252 may be configured to determine the region in which the target pixel 411 is located from among the peripheral region 410, the foveal region 420, and the boundary region 430, based on the position of the target pixel 411 and the boundary region data set stored in the register 250, and to determine the weight to be assigned to the corresponding pixel 412 in the foveal image 401 and the weight to be assigned to the corresponding pixel 413 in the upscaled full-frame image 402, based on the region in which the target pixel 411 is located. In some embodiments, the weight determination circuit 252 may be configured to determine the weight assigned to the corresponding pixel 412 of the foveated image 401 as 100% when the target pixel 411 is located in the foveated region 420 of the foveated image 400. The weight determination circuit 252 may further be configured to determine the weight assigned to the corresponding pixel 412 of the foveated image 401 as 0% when the target pixel 411 is located in the peripheral region 410 of the foveated image 400. The weight determination circuit 252 may further be configured to determine the weight assigned to the corresponding pixel 412 of the foveated image 401 as a value between 0% and 100% when the target pixel 411 is located in the boundary region 430 of the foveated image 400. In an embodiment where the integration of the foveal image 401 with the upscaled full-frame image 402 for rendering the boundary region 430 is performed by alpha blending, the weights assigned to the corresponding pixels 412 of the foveal image 401 may be the alpha values ​​assigned to the corresponding pixels 412 of the foveal image 401.

[0056] In one or more embodiments, the dimensions of the boundary region 430 provided between the peripheral region 410 and the foveal region 420 may be indicated by a boundary region dimension data set stored in register 250. Figure 8 illustrates an exemplary configuration of a foveated image 400 according to one or more embodiments, in which the boundary region 430 is provided between the peripheral region 410 and the foveal region 420. In the illustrated embodiment, the boundary region 430 includes a first region 432-1, a second region 432-2, and a third region 432-3 defined in the shape of a rectangular frame. In one implementation, the first region 432-1 may be defined to completely or substantially enclose the foveal region 420, the second region 432-2 may be defined to completely or substantially enclose the first region 432-1, and the third region 432-3 may be defined to completely or substantially enclose the second region 432-2. In Figure 8, reference numeral 440 indicates the vertical centerline of the foveal image, and reference numeral 450 indicates the horizontal centerline of the foveal image.

[0057] In one or more embodiments, a boundary region dimension data set stored in register 250 indicates the dimensions of the first to third regions 432-1 to 432-3 of the boundary region 430. In the illustrated embodiment, the boundary region dimension data set includes the following dimensions of the first to third regions 432-1 to 432-3. BV1: Vertical width of the upper and lower horizontal portions of Section 1 432-1 BH1: Horizontal width of the vertical portion of Section 1, 432-1 BV2: Vertical width of the horizontal portion at the upper and lower ends of Section 2 432-2 BH2: Horizontal width of the vertical portion of Section 2, 432-2 BV2: Vertical width of the upper and lower horizontal portions of Section 3 432-3 BH3: Horizontal width of the vertical portion of Section 3, 432-3

[0058] The image processing circuit 240 is configured to modify at least one dimension of the boundary region 430 by updating the boundary region dimension dataset stored in the register 250. The modification of the dimensions of the boundary region 430 may be based on control data received from the controller 100A. In one implementation, the boundary region dimension dataset may be rewritable by the controller 100A. By rewriting the boundary region dimension dataset, one or more dimensions of the boundary region 430 can be easily adjusted.

[0059] As described above in relation to Figure 7B, the tonal range of the target pixel in the boundary region 430 may be equal to the weighted sum of the tonal range of the corresponding pixel in the foveal image and the tonal range of the corresponding pixel in the upscaled full-frame image. In such embodiments, the weights assigned to the corresponding pixel in the foveal image and the weights assigned to the corresponding pixel in the upscaled full-frame image are determined based on the position of the target pixel in the foveated image 400. In some implementations, the weights assigned to the corresponding pixel in the foveal image may increase as the distance of the target pixel from the foveal region 420 decreases. By determining the weights in this way, the foveal image and the upscaled full-frame image are blended within the boundary region 430, gradually changing the ratio of the foveal image to the upscaled full-frame image. This can effectively smooth the boundary of the foveal region 420.

[0060] In some embodiments, the weights assigned to corresponding pixels in a foveal image may be based on the region in which the pixel in question is located. Figure 9 illustrates exemplary weights assigned to the tonal range of corresponding pixels in a foveal image according to one or more embodiments.

[0061] In the embodiment shown in Figure 9, when the target pixel is located in the foveal region 420, the weight assigned to the corresponding pixel in the foveal image is 100%. In this case, the gradation of the target pixel is determined to be identical to the gradation of the corresponding pixel in the foveal image. Therefore, the foveal region 420 of the foveated image 400 is rendered based on the foveal image independently of the upscaled full-frame image.

[0062] If the target pixel is located in the first region 432-1 of the boundary region 430 (which is the region closest to the foveal region 420), the weight assigned to the corresponding pixel in the foveal image is X1%. Therefore, the weight assigned to the corresponding pixel in the upscaled full-frame image is 100-X1%.

[0063] Furthermore, if the target pixel is located in the second region 432-2 of the boundary region 430, the weight assigned to the corresponding pixel in the foveal image is X2%, and if the target pixel is located in the third region 432-3 of the boundary region 430, it is X3%. In one implementation, the weight assigned to the corresponding pixel in the foveal image may increase as the target pixel gets closer to the foveal region 420. Therefore, the weight X1 assigned to the corresponding pixel in the foveal image when the target pixel is located in the first region 432-1 is greater than the weight X2 assigned to the corresponding pixel in the foveal image when the target pixel is located in the second region 432-2, and weight X2 is greater than the weight X3 assigned to the corresponding pixel in the foveal image when the target pixel is located in the third region 432-3.

[0064] If the target pixel is located in the peripheral region 410, the weight assigned to the corresponding pixel in the foveal image is 0%. In such cases, the gradation of the target pixel is identical to that of the corresponding pixel in the upscaled full-frame image. Therefore, the peripheral region 410 of the foveated image 400 is rendered based only on the upscaled full-frame image (and not on the foveal image).

[0065] Figure 10 shows an exemplary foveated image 205A, obtained by superimposing the foveal image 206A onto the upscaled full-frame image 207A using weights assigned to the pixels of the foveal image 206A and the upscaled full-frame image 207A, as illustrated in Figure 9, according to one or more embodiments. In the exemplary foveated image 205A, the boundary of the foveal image 206A is smoothed to mitigate artifacts at the boundary.

[0066] The image processing circuit 240 may be further configured to perform additional image processing on the foveated image rendered by the integration circuit 246, and to transfer the image generated by the image processing to the drive circuit 260. The image processing performed by the image processing circuit 240 may include color adjustment, gamma conversion, overdrive, or other image processing.

[0067] Figure 11 illustrates exemplary weights assigned to pixels in a foveal image according to several other embodiments. In the illustrated embodiments, the weight assigned to the foveal image increases gradually (e.g., monotonically) with decreasing distance from the foveal region 420 of the target pixel, while the weight assigned to the upscaled full-frame image increases gradually (e.g., monotonically) with decreasing distance from the peripheral region 410 of the target pixel. Although Figure 11 illustrates that the weight assigned to the grayscale of a pixel in the foveal image increases linearly with decreasing distance from the foveal region 420 of the target pixel, the weight assigned to the grayscale of a pixel in the foveal image may increase non-linearly.

[0068] Figure 12 illustrates an exemplary configuration of a foveated image 500 according to another embodiment. The foveated image 500 includes a peripheral region 510, a foveal region 520, and a boundary region 530 provided between the peripheral region 510 and the foveal region 520. In Figure 12, reference numeral 540 indicates the vertical centerline of the foveal image, and reference numeral 550 indicates the horizontal centerline of the foveal image. In the illustrated embodiment, the corners of the foveal region 520 and the boundary region 530 are rounded. Each of the four corners of the inner boundary of the boundary region 530 (i.e., the four corners of the foveal region 520) has the shape of a quadrant circumference of radius AR, and adjacent corners are connected by line segments. Each of the four corners of the outer boundary of the boundary region 530 also has the shape of a quadrant circumference.

[0069] In the illustrated embodiment, the boundary region 530 includes a first region 532-1, a second region 532-2, and a third region 532-3. In one implementation, the first region 532-1 is annular and surrounds the foveal region 520, while the second region 532-2 and the third region 532-3 are annular and surround the first region 532-1 and the second region 532-2, respectively. Meanwhile, the first region 532-1, the second region 532-2, and the third region 532-3 each have a constant width. Figure 12 illustrates three regions 532-1, 532-2, and 532-3, but the boundary region 530 may include two, four, or more regions.

[0070] In one or more embodiments, the boundary area dimension data set stored in register 250 indicates the dimensions of the boundary area 530, including the dimensions of the first to third areas 532-1 to 532-3. In the illustrated embodiment, the boundary area dimension data set includes the following dimensions: AR: Radius of the corner of the inner boundary of boundary region 530 SH: Distance between the vertical centerline 540 and the inner boundary corner of the boundary region 530. SV: Distance between the horizontal centerline 540 and the inner boundary corner of the boundary region 530. BR1: Width of Section 1, 532-1 BR2: Width of Section 2, 532-2 BR3: Width of Section 3, 532-3

[0071] The image processing circuit 240 is configured to modify at least one dimension of the boundary region 530 by updating the boundary region dimension dataset stored in the register 250. The modification of the dimensions of the boundary region 530 may be based on control data received from the controller 100A. In some implementations, the boundary dimension dataset is rewritable by the controller 100A. By rewriting the boundary region dimension dataset, one or more dimensions of the boundary region 530 can be easily adjusted.

[0072] Figure 13 illustrates exemplary weights assigned to pixels in a foveal image according to one or more embodiments. In the illustrated embodiments, when the target pixel is located in the foveal region 520, the weight assigned to the corresponding pixel in the foveal image is 100%. Therefore, the grayscale of the target pixel is identical to the grayscale of the corresponding pixel in the foveal image. Thus, the foveal region 520 of the foveated image 500 is rendered based solely on the foveal image (and not on the upscaled full-frame image).

[0073] When the target pixel is located in the first region 532-1, which is closest to the foveal region 520 in the boundary region 530, the weight assigned to the corresponding pixel in the foveal image is Y1%. Therefore, the weight assigned to the corresponding pixel in the upscaled full-frame image is 100-Y1%.

[0074] Furthermore, when the target pixel is located in the second region 532-2 of the boundary region 530, the weight assigned to the corresponding pixel in the foveal image is Y2%, and when the target pixel is located in the third region 532-3 of the boundary region 530, the weight is Y3%. When the target pixel is located in the first region 532-1, the weight Y1 assigned to the corresponding pixel in the foveal image is greater than the weight Y2 assigned to the corresponding pixel in the foveal image when the target pixel is located in the second region 532-2, and weight Y2 is greater than the weight Y3 assigned to the corresponding pixel in the foveal image when the target pixel is located in the third region 532-3.

[0075] When the target pixel is located in the peripheral region 510, the weight assigned to the corresponding pixel in the foveal image is 0%. Therefore, the gradation of the target pixel is identical to the gradation of the corresponding pixel in the upscaled full-frame image. Thus, the peripheral region 510 of the foveated image 500 is rendered based only on the upscaled full-frame image (and not on the foveal image).

[0076] In one or more embodiments, the control data transmitted from the controller 100A to the display driver 200A may include boundary area control data, and the display driver 200A may be configured to adjust the configuration of the boundary area 430 or 530 based on the boundary area control data. In some implementations, the boundary area control data may specify at least one dimension of the boundary area 430 or 530.

[0077] In some embodiments, as illustrated in Figure 14, the register 250 may be configured to store multiple boundary region dimension datasets, each defining a boundary region with a different configuration and / or different dimensions. In such embodiments, the boundary region control data may specify one of the boundary region dimension datasets, and the weight determination circuit 252 may assign various weights to the pixels of the foveal image and the pixels of the upscaled full-frame image based on the specified boundary region dimension dataset.

[0078] In the illustrated embodiment, register 250 is configured to store boundary area dimension data sets 254-1, 254-2, and 254-3, each defining a boundary area of ​​a different size. Boundary area dimension data set 254-1 defines a "large" size boundary area, boundary area dimension data set 254-2 defines a "medium" size boundary area, and boundary area dimension data set 254-3 defines a "small" size boundary area.

[0079] In some embodiments, boundary region control data may specify the size of the boundary region as "large," "medium," or "small." In such embodiments, register 250 may be configured to select one of the boundary region dimension datasets 254-1, 254-2, and 254-3 based on the size of the boundary region specified by the boundary region control data, and to supply the selected boundary region dimension dataset to the weight determination circuit 252. The weight determination circuit 252 may be configured to determine the weights to be assigned to the pixels of the foveal image and the weights to be assigned to the upscaled full-frame image based on the selected boundary region dimension dataset received from register 250.

[0080] Method 1500 in Figure 15 illustrates exemplary steps for operating a foveated rendering-compatible display driver (e.g., display driver 200A shown in Figure 7A) according to one or more embodiments. Note that one or more steps shown in Figure 15 may be omitted, repeated, and / or performed in an order different from that shown in Figure 15. Also note that two or more steps may be performed simultaneously.

[0081] The method includes, in step 1502, receiving a full-frame image and a foveal image from an external source to the display driver (e.g., controller 100A as shown in Figure 5). The method further includes, in step 1504, upscaling the full-frame image. The method further includes, in step 1506, rendering a foveated image (e.g., foveated images 400 and 500) from the upscaled full-frame image and the foveal image. The foveated image includes a foveal region based on the foveal image (e.g., foveal regions 420 and 520), a peripheral region based on the upscaled full-frame image (e.g., peripheral regions 410 and 510 in Figures 4C, 8 and 12), and a boundary region based on the foveal image and the upscaled full-frame image (e.g., boundary regions 430 and 530). The boundary region is located between the foveal region and the peripheral region. The method further includes driving a display panel (e.g., display panel 300) based on a foveated image in step 1508.

[0082] Although many embodiments have been described, those skilled in the art who are interested in this disclosure will likely find that other embodiments within the scope are possible. Therefore, the technical scope of the present invention should be limited only by the appended claims.

Claims

1. A display driver, interface circuitry configured to receive a full-frame image and a foveated image from a source external to the display driver; image processing circuitry configured to upscale the full-frame image and render a foveated image from the upscaled full-frame image and the foveated image; a drive circuit configured to drive a display panel using the foveated image; Equipped with The foveated image is a foveal region based on the foveal image; a surrounding region based on the upscaled full-frame image; a boundary region based on the foveated image and the upscaled full-frame image; Including, The boundary region is located between the foveal region and the peripheral region. Display driver.

2. Rendering the foveated image includes determining a gray level of a target pixel in the boundary region based on a weighted sum of a gray level of a first pixel of the foveated image and a gray level of a second pixel of the upscaled full-frame image.

2. The display driver according to claim 1.

3. The boundary region is a first area surrounding the foveal region; a second area surrounding the first area; Including, Rendering the foveated image comprises: determining a gray level of a first pixel of interest in the first area based on a first weighted sum of a gray level of a first pixel of the foveated image and a gray level of a second pixel of the upscaled full-frame image; determining a gray level of a second pixel of interest in the second area based on a second weighted sum of a gray level of a third pixel of the foveated image and a gray level of a fourth pixel of the upscaled full-frame image; Including, a first weight is assigned to the first pixel in the first weighted sum, and a second weight is assigned to the third pixel in the second weighted sum; the first weight is greater than the second weight; 2. The display driver according to claim 1.

4. further comprising a register configured to store a bounding area dimension data set indicative of one or more dimensions of the bounding area; The bounding region is rendered based on the bounding region dimension dataset.

2. The display driver according to claim 1.

5. further comprising register circuitry configured to store a plurality of bounding area dimension data sets, each data set indicating one or more dimensions of the bounding area; The image processing circuit unit further receiving control data from the source via the interface circuitry; configured to select one of the plurality of bounding area dimension data sets based on the control data; The bounding region is rendered based on the selected one of the plurality of bounding region dimension data sets.

2. The display driver according to claim 1.

6. a source configured to render a full-frame image and a foveated image; A display panel; a display driver configured to receive the full-frame image and the foveated image from the source, upscale the full-frame image, render a foveated image from the upscaled full-frame image and the foveated image, and drive the display panel with the foveated image; Equipped with The foveated image is a foveal region based on the foveal image; a surrounding region based on the upscaled full-frame image; a boundary region based on the foveated image and the upscaled full-frame image; Including, The boundary region is located between the foveal region and the peripheral region. Display system.

7. Rendering the foveated image includes determining a gray level of a target pixel in the boundary region based on a weighted sum of a gray level of a first pixel of the foveated image and a gray level of a second pixel of the upscaled full-frame image. The display system of claim 6.

8. The boundary region is a first area surrounding the foveal region; a second area surrounding the first area; Including, Rendering the foveated image comprises: determining a gray level of a first pixel of interest in the first area based on a first weighted sum of a gray level of a first pixel of the foveated image and a gray level of a second pixel of the upscaled full-frame image; determining a gray level of a second pixel of interest in the second area based on a second weighted sum of a gray level of a third pixel of the foveated image and a gray level of a fourth pixel of the upscaled full-frame image; Including, a first weight is assigned to the first pixel in the first weighted sum, and a second weight is assigned to the third pixel in the second weighted sum; the first weight is greater than the second weight; The display system of claim 6.

9. 1. A method of operating a display driver, comprising: receiving a full-frame image and a foveated image from a source external to the display driver; upscaling the full-frame image; Rendering a foveated image from the upscaled full-frame image and the foveated image; driving a display panel using the foveated image; Including, The foveated image is a foveal region based on the foveal image; a surrounding region based on the upscaled full-frame image; a boundary region based on the foveated image and the upscaled full-frame image; Including, The boundary region is located between the foveal region and the peripheral region. method.

10. Rendering the foveated image includes determining a gray level of a target pixel in the boundary region based on a weighted sum of a gray level of a first pixel of the foveated image and a gray level of a second pixel of the upscaled full-frame image.

10. The method of claim 9.

Images