AR / VR image display method, AR / VR image display device, and computer program product

Abstract

An augmented reality / virtual reality (AR / VR) image display method is provided. The AR / VR method includes: determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel.

Description

Technical Field

[0001] The present invention relates to display technology, and more particularly, to an augmented reality / virtual reality (AR / VR) image display method, an AR / VR image display device, and a computer program product. Background Art

[0002] With the continuous development of display technology, ultra-high-resolution small screens are increasingly favored by major manufacturers and various types of smart display products. In particular, for AR / VR image display, ultra-high-resolution image display is required. Summary of the invention

[0003] In one aspect, the present disclosure provides an augmented reality / virtual reality (AR / VR) image display method, comprising: determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel.

[0004] Optionally, the method is used for AR image display; the method also includes: determining the size and position of a first virtual object to be displayed in a previous adjacent frame image; determining the size and position of a second virtual object to be displayed in a current frame image, wherein the previous adjacent frame image and the current frame image are two consecutive frame images; determining a first regular polygon display area in which the first virtual object in the previous adjacent frame image will be displayed, the first regular polygon display area being smaller than a total display area; determining a second regular polygon display area in which the second virtual object in the current frame image will be displayed, the second regular polygon display area being smaller than the total display area; determining a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, is equal to or larger than the first regular polygon display area, and is equal to or larger than the second regular polygon display area; and displaying a composite image in an area limited to the composite regular polygon display area in the current frame image.

[0005] Optionally, the second regular polygon display area is used as the reduced display area to determine the second field of view; the reduced image resolution is used as the current image resolution of the composite image in the current frame image; and the intermediate image is rendered based on the current image resolution and the second field of view.

[0006] Optionally, displaying the composite image in the current frame image includes providing gate scan signals and data signals restricted to a limited subset of sub-pixels among a plurality of sub-pixels of the display panel.

[0007] Optionally, the composite image includes the intermediate image in a first area of ​​the composite regular polygon display area and a background image in a second area of ​​the composite regular polygon display area, the first area corresponds to the second regular polygon display area, and the second area does not overlap with the first area; the sub-pixels of the finite subset correspond to the composite regular polygon display area; the data signal of the intermediate image is provided to the sub-pixels corresponding to the first area; and the background data signal is provided to the sub-pixels corresponding to the second area.

[0008] Optionally, the sub-pixels of the finite subset include sub-pixels whose grayscale is greater than a threshold; and the sub-pixels of the finite subset correspond to a finite area of ​​the synthetic regular polygon display area.

[0009] Optionally, rendering the intermediate image includes generating a virtual data signal including information about the size and position of the second virtual object and the resolution of the current image; and the method also includes sending the virtual data signal together with the data signal of the intermediate image to the integrated circuit; and buffering the virtual data signal in the integrated circuit for use in the next adjacent frame image.

[0010] Optionally, the first regular polygon display area is the minimum polygon display area capable of displaying the first virtual object in the previous adjacent frame image; the second regular polygon display area is the minimum polygon display area capable of displaying the second virtual object in the current frame image; and the composite regular polygon display area is the minimum polygon display area including the first regular polygon display area and the second regular polygon display area.

[0011] Optionally, the first regular polygon display area and the second regular polygon display area overlap each other.

[0012] Optionally, the first regular polygon display area, the second regular polygon display area and the synthesized regular polygon display area have the same type of polygon shape.

[0013] Optionally, the first regular polygon display area is a first rectangular display area; the second regular polygon display area is a second rectangular display area; and the synthetic regular polygon display area is a synthetic rectangular display area.

[0014] Optionally, the method is used for VR image display; the method also includes: displaying the intermediate image in the reduced display area; and displaying a background image in an area outside the reduced display area of ​​the total display area.

[0015] Optionally, displaying the intermediate image includes providing gate scan signals and data signals to sub-pixels that are restricted to a limited subset of multiple sub-pixels of the display panel; the sub-pixels of the limited subset correspond to the reduced display area; and displaying the background image includes directly assigning grayscale values ​​to sub-pixels in the area outside the reduced display area of ​​the total display area by the integrated circuit.

[0016] Optionally, the reduced display area includes a high-resolution area and a low-resolution area; and the limited subset of sub-pixels includes all sub-pixels corresponding to the high-resolution area and sub-pixels corresponding only to alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area.

[0017] Optionally, the AR / VR image display method also includes: obtaining the coordinates of the gaze point; based on the coordinates of the gaze point, rendering a high-definition image for display in the sub-pixels corresponding to the high-resolution area; rendering a low-definition image for display in the alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area; processing the high-definition image and the low-definition image using an image processing algorithm to generate a processed high-definition image and a processed low-definition image; and sending the processed high-definition image and the processed low-definition image to the integrated circuit; wherein the intermediate image includes the processed high-definition image and the processed low-definition image.

[0018] Optionally, the AR / VR image display method also includes: stretching the processed low-definition image by the integrated circuit to generate a stretched low-definition image; splicing the stretched low-definition image with the processed high-definition image by the integrated circuit to generate a spliced ​​image; and displaying the spliced ​​image.

[0019] Optionally, the image processing algorithm is one selected from the group consisting of an image enhancement algorithm and a local dimming algorithm.

[0020] Optionally, rendering the intermediate image includes generating a virtual data signal including information of the size and position of the reduced display area and the reduced image resolution; and the method also includes sending the virtual data signal together with the data signal of the intermediate image to the integrated circuit.

[0021] On the other hand, the present disclosure provides an augmented reality / virtual reality (AR / VR) image display device, comprising: a display panel; an integrated circuit connected to the display panel; a memory; and one or more processors; wherein the memory and the one or more processors are connected to each other; and the memory stores computer executable instructions for controlling the one or more processors to perform the following operations: determine a first field of view corresponding to a total display area; determine a second field of view corresponding to a reduced display area; obtain a screen resolution corresponding to the first field of view; determine a reduced image resolution based on the first field of view, the second field of view and the screen resolution; render an intermediate image based on the reduced image resolution and the second field of view; and send a data signal of the intermediate image to the integrated circuit connected to the display panel.

[0022] In another aspect, the present disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to: determine a first field of view corresponding to a total display area; determine a second field of view corresponding to a reduced display area; obtain a screen resolution corresponding to the first field of view; determine a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the reduced image resolution and the second field of view; and send a data signal of the intermediate image to an integrated circuit connected to a display panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The following drawings are merely examples according to various disclosed embodiments for illustrative purposes and are not intended to limit the scope of the invention.

[0024] Figure 1 An augmented reality / virtual reality (AR / VR) image display method according to some embodiments of the present disclosure is shown.

[0025] Figure 2 The field of view in the AR / VR image display method in some embodiments of the present disclosure is shown.

[0026] Figure 3 An AR image display method according to some embodiments of the present disclosure is shown.

[0027] Figure 4 A synthetic regular polygon display area in some embodiments according to the present disclosure is shown.

[0028] Figure 5 An AR image display method according to some embodiments of the present disclosure is shown.

[0029] Figure 6Composite images in accordance with some embodiments of the present disclosure are shown.

[0030] Figure 7 Composite images in accordance with some embodiments of the present disclosure are shown.

[0031] Figure 8 A VR image display method according to some embodiments of the present disclosure is shown.

[0032] Fig. 9 A VR image display method according to some embodiments of the present disclosure is shown.

[0033] Fig.10 A VR image display method according to some embodiments of the present disclosure is shown.

[0034] Fig.11 An intermediate image and a background image in a VR image display method in some embodiments of the present disclosure are shown.

[0035] Fig.12 A VR image display method according to some embodiments of the present disclosure is shown.

[0036] Fig.13 is a schematic diagram of the structure of a device in some embodiments of the present disclosure.

[0037] Fig.14 is a schematic diagram showing the structure of an AR / VR image display device in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] The present disclosure will now be described in more detail with reference to the following examples. It should be noted that the following description of some of the embodiments presented herein is for illustration and description purposes only. It is not intended to be exhaustive or limited to the precise form disclosed.

[0039] The present disclosure provides, among other things, an augmented reality / virtual reality (AR / VR) image display method, an AR / VR image display device, and a computer program product, which substantially eliminate one or more problems caused by limitations and disadvantages of the prior art. In one aspect, the present disclosure provides an AR / VR image display method. In some embodiments, the AR / VR image display method includes: determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel.

[0040] Figure 1An augmented reality / virtual reality (AR / VR) image display method according to some embodiments of the present disclosure is shown. Figure 1 , in some embodiments the AR / VR image display method includes: determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel. As used herein, the term "augmented reality" (AR) is defined as a real-time direct or indirect view of a physical, real-world environment, elements of which are enhanced (or supplemented) by computer-generated sensory input (such as sound, video, graphics, or GPS data). It relates to a more general concept known as mediated reality, in which the view of reality is modified by a computer (perhaps even attenuated rather than enhanced). As used herein, the term "virtual reality" or VR is defined as a computer-simulated environment that can simulate physical presence in places in the real world or imagined world. Virtual reality can recreate sensory experiences, including virtual taste, vision, smell, sound, touch, etc. Many traditional VR systems use near-eye displays to present 3D virtual environments.

[0041] Figure 2 The field of view in the AR / VR image display method in some embodiments of the present disclosure is shown. Figure 2 The viewing frustum VF or the visible area of ​​the three-dimensional space is shown. The real world setting may involve a camera, while the virtual world setting may involve a viewpoint VP (often referred to as a virtual camera). The field of view FOV is usually discussed in relation to the viewing frustum VF. Figure 2 , describing the rectangular viewport, so Figure 2 The viewing frustum in is a truncated four-sided (e.g., rectangular) pyramid. The viewport may have various other suitable shapes, such as a circle, and thus the viewing frustum VF may have other suitable shapes, such as a cone. The field of view FOV includes a vertical field of view VFOV and a horizontal field of view HFOV. The boundaries of the viewing frustum VF are defined by the vertical field of view VFOV, the horizontal field of view HFOV, the near limit, and the far limit. The near limit is defined by the near clipping plane CPn of the viewing frustum VF, and the far limit is defined by the far clipping plane CPf of the viewing frustum VF. The viewing frustum VF may be further defined by other parameters, such as position and orientation. The viewing frustum VF in the virtual world typically corresponds to the field of view of a real camera in the real world.

[0042] In some embodiments, the method is used for AR image display. Figure 3The AR image display method in some embodiments of the present disclosure is shown. Figure 3 In some embodiments, the AR / VR image display method further includes: determining the size and position of a first virtual object to be displayed in a previous adjacent frame image; determining the size and position of a second virtual object to be displayed in a current frame image, wherein the previous adjacent frame image and the current frame image are two consecutive frame images; determining a first regular polygon display area in which the first virtual object in the previous adjacent frame image will be displayed, wherein the first regular polygon display area is smaller than the total display area; determining a second regular polygon display area in which the second virtual object in the current frame image will be displayed, wherein the second regular polygon display area is smaller than the total display area; determining a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, is equal to or larger than the first regular polygon display area, and is equal to or larger than the second regular polygon display area; and displaying the composite image in an area limited to the composite regular polygon display area in the current frame image.

[0043] In some embodiments, the AR image display method includes sequentially displaying multiple frames of images. The multiple frames of images include a previous adjacent frame image, a current frame image, and a next adjacent frame image. In one example, the AR image display method includes displaying N frames of images, where N is an integer greater than 1. Optionally, the AR image display method includes determining the size and position of a first virtual object to be displayed in the (n-1)th frame image, where n is an integer and 1<n<N; determining the size and position of a second virtual object to be displayed in the (n)th frame image, where the (n-1)th frame image and the (n)th frame image are two consecutive frame images; determining a first regular polygon display area in which the first virtual object in the n-1th frame image will be displayed, the first regular polygon display area being smaller than the total display area; determining a second regular polygon display area in which the second virtual object in the nth frame image will be displayed, the second regular polygon display area being smaller than the total display area; determining a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, is equal to or larger than the first regular polygon display area, and is equal to or larger than the second regular polygon display area; and in the (n)th frame image, displaying the composite image in an area limited to the composite regular polygon display area.

[0044] In each frame image (e.g., current frame image, (n)th frame image, previous adjacent frame image, (n-1)th frame image, next adjacent frame image, (n+1)th frame image), the AR image display method includes calculating the position of a virtual object (e.g., a three-dimensional virtual object) and the position of a viewpoint (e.g., a virtual camera). Based on the position of the virtual object (e.g., the first virtual object in the previous adjacent frame image, or the second virtual object in the current frame image), the method also includes calculating the size of the virtual object, and rendering the virtual object based on the size of the virtual object.

[0045] Figure 4 FIG. 4 shows a synthetic regular polygon display area according to some embodiments of the present disclosure. Figure 4 In some embodiments, the method includes: determining a first regular polygon display area PDA1, in which a first virtual object VO1 in a previous adjacent frame image is to be displayed, and the first regular polygon display area PDA1 is smaller than a total display area TDA; determining a second regular polygon display area PDA2, in which a second virtual object VO2 in a current frame image is to be displayed, and the second regular polygon display area PDA2 is smaller than the total display area TDA; determining a composite regular polygon display area CPDA including the first regular polygon display area PDA1 and the second regular polygon display area PDA2, wherein the composite regular polygon display area CPDA is smaller than the total display area TDA, is equal to or larger than the first regular polygon display area PDA1, and is equal to or larger than the second regular polygon display area PDA2. In the current frame image, the composite image is displayed in an area limited to the composite regular polygon display area CPDA.

[0046] In some embodiments, the first regular polygon display area PDA1 is a minimum polygon display area in which the first virtual object VO1 in the first frame image can be displayed. The second regular polygon display area PDA2 is a minimum polygon display area in which the second virtual object VO2 in the second frame image can be displayed. The composite regular polygon display area CPDA is a minimum polygon display area including the first regular polygon display area PDA1 and the second regular polygon display area PDA2.

[0047] In some embodiments, the first regular polygon display area PDA1 is the minimum polygon display area in which the first virtual object VO1 in the first frame image can be displayed plus a first buffer area surrounding the minimum polygon display area. The second regular polygon display area PDA2 is the minimum polygon display area in which the second virtual object VO2 in the second frame image can be displayed plus a second buffer area surrounding the minimum polygon display area. The composite regular polygon display area CPDA is a polygon display area including the first regular polygon display area PDA1 and the second regular polygon display area PDA2, with or without an additional buffer area.

[0048] Alternatively, the first regular polygon display area PDA1 and the second regular polygon display area PDA2 overlap each other.

[0049] Alternatively, the first regular polygon display area PDA1 and the second regular polygon display area PDA2 do not substantially overlap each other.

[0050] In some embodiments, the first regular polygon display area PDA1, the second regular polygon display area PDA2, and the composite regular polygon display area CPDA have the same type of polygon shape. Figure 4 As shown, the first regular polygon display area PDA1 is a first rectangular display area; the second regular polygon display area PDA2 is a second rectangular display area; and the composite regular polygon display area CPDA is a composite rectangular display area.

[0051] Figure 5 The AR image display method in some embodiments of the present disclosure is shown. Figure 5 In some embodiments, before displaying the synthesized image in the current frame image, the AR image display method further includes determining a first field of view corresponding to the total display area; determining a second field of view corresponding to the second regular polygon display area; obtaining a screen resolution corresponding to the first field of view; determining a current image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the current image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to the display panel to display the synthesized image. In some embodiments, the second regular polygon display area is used as a reduced display area for determining the second field of view. The reduced image resolution is used as the current image resolution of the synthesized image in the current frame image. The intermediate image is rendered based on the current image resolution and the second field of view.

[0052] In this AR image display method, instead of rendering an image based on a first field of view corresponding to the total display area of ​​the display panel, only an intermediate image based on the current image resolution and a second field of view corresponding to the second regular polygon display area is rendered. In this AR image display method, the computing resources required to render an image for displaying a virtual object can be greatly reduced. In addition, the transmission bandwidth required to send the rendered image to the integrated circuit of the display panel can be significantly reduced. The inventors of the present disclosure have found that the AR image display can be significantly improved by this method, especially for high-resolution AR image display.

[0053] In some embodiments, the current image resolution (e.g., reduced image resolution) is determined based on the first field of view, the second field of view, and the screen resolution. In one example, the reduced image resolution is calculated based on equations (1) and (2):

[0054] res_me.x = res_1.x * fov_me.x / fov_1.x (1); and

[0055] res_me.y=res_1.y*fov_me.x / fov_1.y (2);

[0056] Where res_me.x represents the reduced horizontal image resolution of the current frame image, res_me.y represents the reduced vertical image resolution of the current frame image, fov_me.x represents the horizontal field of view of the second field of view, fov_me.y represents the vertical field of view of the second field of view, fov_1.x represents the horizontal field of view of the first field of view, and fov_1.y represents the vertical field of view of the first field of view.

[0057] In some embodiments, when rendering the intermediate image, the method further includes: generating a virtual data signal including information about the size and position of the second virtual object and the current image resolution. The virtual data signal is sent to the integrated circuit of the display panel together with the data signal of the intermediate image. In one example, the intermediate image includes a data signal corresponding to a row of virtual sub-pixels for containing information about the size and position of the second virtual object and the current image resolution. The virtual data signal for the current frame image is cached in the integrated circuit, and then this information is used to display the image in the next adjacent frame image (e.g., the (n+1)th frame image). As above, the amount of computing resources and transmission bandwidth required for the AR image display method are greatly reduced.

[0058] In some embodiments, displaying a composite image in a current frame image includes providing gate scan signals and data signals for sub-pixels of a limited subset of a plurality of sub-pixels of a display panel. In sub-pixels other than the limited subset of sub-pixels, the method does not involve scanning sub-pixels. For example, only in a limited subset of sub-pixels, a gate scan signal is provided to scan the sub-pixels row by row (turning on or off the transistors therein) to allow input of data signals. Since only a limited subset of sub-pixels is scanned in this AR image display method, the refresh rate of the display panel can be greatly improved, and an excellent AR display effect can be achieved.

[0059] Figure 6 A composite image according to some embodiments of the present disclosure is shown. Figure 6In some embodiments, the composite image includes an intermediate image in a first region R1 of a composite regular polygon display area CPDA and a background image in a second region R2 of the composite regular polygon display area CPDA, the first region R1 corresponds to the first regular polygon display area PDA2, and the second region R2 does not overlap with the first region R1. In one example, a dark image is displayed in the second region R2, and the sub-pixels corresponding to the second region R2 have a grayscale value equal to or close to zero. Images of various other colors may be displayed in the second region R2, for example, a grayscale image, a blue image, a red image, etc. Optionally, all sub-pixels corresponding to the second region R2 have a substantially uniform grayscale. By displaying the background image in the second region R2, the previous adjacent frame image can be reset.

[0060] In some embodiments, the scanned limited subset of sub-pixels corresponds to the synthetic regular polygon display area CPDA. The data signal of the intermediate image is provided to the sub-pixels corresponding to the first region R1. Optionally, the background data signal (e.g., a data signal of substantially uniform grayscale) is provided to the sub-pixels corresponding to the second region R2.

[0061] Figure 7 A composite image according to some embodiments of the present disclosure is shown. Figure 7 In some embodiments, only the sub-pixels in the white line sub-region of the synthetic regular polygon display area CPDA are scanned, and sub-pixels other than the sub-pixels in the white line sub-region are not scanned (for example, gate scanning signals are not received to turn on or off transistors therein). Therefore, in some embodiments, the AR image display method includes displaying a synthetic image in a current frame image, including providing gate scanning signals and data signals for sub-pixels that are limited to a limited subset of multiple sub-pixels of a display panel, wherein the limited subset of sub-pixels includes sub-pixels having a grayscale greater than a threshold value (for example, greater than 5, greater than 10, greater than 15, or greater than 20). The limited subset of sub-pixels corresponds to a limited area RL of the synthetic regular polygon display area CPDA. By having such more precise sub-pixel scanning control, the refresh rate of the display panel can be further improved and power consumption can be reduced.

[0062] In some embodiments, the method is used for VR image display. Figure 1 In some embodiments, VR image display includes determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel.

[0063] Figure 8The VR image display method in some embodiments of the present disclosure is shown. Figure 8 In some embodiments, a VR image display method includes obtaining the coordinates of a gaze point (e.g., by a gaze tracking device); based on the coordinates of the gaze point, rendering a high-definition image (e.g., by a processor) for display in sub-pixels corresponding to a high-resolution area; rendering a low-definition image (e.g., by a processor) for display in alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area; processing (e.g., by a processor) the high-definition image and the low-definition image using an image processing algorithm to generate a processed high-definition image and a processed low-definition image; and sending the processed high-definition image and the low-definition image to an integrated circuit. The intermediate image includes a processed high-definition image and a processed low-definition image. The integrated circuit is configured to stretch the processed low-definition image to generate a stretched low-definition image when receiving the intermediate image; and stitch the stretched low-definition image and the processed high-definition image together to generate a stitched image. The display panel is configured to display the stitched image. Various appropriate image processing algorithms can be used to process the high-definition image and the low-definition image. Examples of image processing algorithms include image enhancement algorithms and local dimming algorithms.

[0064] Fig. 9 The VR image display method in some embodiments of the present disclosure is shown. Fig. 9 In some embodiments, once the coordinates of the gaze point are obtained, the high-definition image and the low-definition image can be rendered. In one example, the density of sub-pixels configured to emit light in the high-definition image and the low-definition image, respectively, is substantially the same. However, the low-definition image in the VR image display is not the focus of the VR experience. Therefore, by configuring the sub-pixels in the low-definition image to emit light with the same density as the sub-pixels in the high-definition image, computing resources and transmission bandwidth are wasted.

[0065] Fig.10 The VR image display method in some embodiments of the present disclosure is shown. Fig.10 In order to save computing resources and transmission bandwidth, only alternate rows of sub-pixels in the low-resolution image are scanned (e.g., the alternate rows of sub-pixels are provided with gate scan signals and data signals). Therefore, in some embodiments, displaying the intermediate image includes providing gate scan signals and data signals to sub-pixels that are limited to a limited subset of the plurality of sub-pixels of the display panel. Fig.10As shown, the limited subset of sub-pixels includes all sub-pixels corresponding to the high-resolution area and only sub-pixels corresponding to alternate rows of sub-pixels in the sub-pixels corresponding to the low-resolution area. Sub-pixels in other rows in the low-resolution area are not scanned (e.g., gate scanning signals and data signals are not provided). By scanning fewer (e.g., almost half) sub-pixels in the low-resolution area, the refresh rate of the display panel can be greatly improved.

[0066] Fig.11 The figure shows the intermediate image and the background image in the VR image display method in some embodiments of the present disclosure. Fig.11 In order to further reduce the computing resources and transmission bandwidth requirements in VR image display, in some embodiments, the VR image display method includes displaying the intermediate image IM in a reduced display area; and displaying the background image BI in an area outside the reduced display area of ​​the total display area. In one example, Fig.11 The VR image display shown is a VR image display in the cinema mode.

[0067] Fig.12 The VR image display method in some embodiments of the present disclosure is shown. Fig.12 , in cinema mode, the method includes in a reduced display area (in Fig.12 In some embodiments, displaying the intermediate image includes providing gate scan signals and data signals to sub-pixels of a limited subset of a plurality of sub-pixels of the display panel. Optionally, the sub-pixels of the limited subset correspond to the reduced display area. Optionally, displaying the background image includes assigning grayscale values ​​directly to sub-pixels in the area outside the reduced display area of ​​the total display area by the integrated circuit. In one example, a grayscale value equal to or close to zero is assigned to sub-pixels in the area outside the reduced display area of ​​the total display area, thereby forming a background image. In another example, images of various other colors may be displayed in the area outside the reduced display area of ​​the total display area, for example, a grayscale image, a blue image, a red image, etc. Optionally, all sub-pixels corresponding to the area outside the reduced display area of ​​the total display area are assigned substantially uniform grayscale values. By displaying the background image in the area outside the reduced display area of ​​the total display area, a cinema mode may be achieved.

[0068] Reference again Fig.12In some embodiments, the reduced display area includes a high-resolution area and a low-resolution area. In order to save computing resources and transmission bandwidth, only alternating rows of sub-pixels in the low-resolution area are scanned (e.g., provided with gate scan signals and data signals). Therefore, in some embodiments, displaying the intermediate image includes providing gate scan signals and data signals that are limited to a limited subset of the plurality of sub-pixels corresponding to the reduced display area. Fig.12 As shown, the limited subset of sub-pixels includes all sub-pixels corresponding to the high-resolution area and only sub-pixels corresponding to alternate rows of sub-pixels in the sub-pixels corresponding to the low-resolution area. Sub-pixels in other rows in the low-resolution area are not scanned (e.g., gate scanning signals and data signals are not provided). By scanning fewer (e.g., almost half) sub-pixels in the low-resolution area, the refresh rate of the display panel can be greatly improved.

[0069] On the other hand, the present disclosure provides an AR / VR image display device. Fig.13 is a schematic diagram of the structure of a device in some embodiments of the present disclosure. Fig.13 In some embodiments, the device includes a central processing unit (CPU) configured to perform actions according to computer executable instructions stored in ROM or RAM. Optionally, data and programs required by the computer system are stored in RAM. Optionally, the CPU, ROM and RAM are electrically connected to each other via a bus. Optionally, the input / output interface is electrically connected to the bus.

[0070] Fig.14 Schematic diagram showing the structure of an AR / VR image display device in some embodiments of the present disclosure. Fig.14 In some embodiments, an AR / VR image display device includes a display panel DP; an integrated circuit IC connected to the display panel DP; a memory M; and one or more processors P. The memory M and the one or more processors P are connected to each other. In some embodiments, the memory M stores computer-executable instructions for controlling the one or more processors P to determine a first field of view corresponding to a total display area; determine a second field of view corresponding to a reduced display area; obtain a screen resolution corresponding to the first field of view; determine a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the reduced image resolution and the second field of view; and send a data signal of the intermediate image to the integrated circuit connected to the display panel DP.

[0071] In some embodiments, the AR / VR image display device is an AR image display device. The memory M stores computer executable instructions for controlling one or more processors P to determine the size and position of a first virtual object to be displayed in a previous adjacent frame image; determine the size and position of a second virtual object to be displayed in a current frame image, wherein the previous adjacent frame image and the current frame image are two consecutive frame images; determine a first regular polygon display area where the first virtual object in the previous adjacent frame image will be displayed, the first regular polygon display area being smaller than the total display area; determine a second regular polygon display area where the second virtual object in the current frame image will be displayed, the second regular polygon display area being smaller than the total display area; determine a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, equal to or larger than the first regular polygon display area, and equal to or larger than the second regular polygon display area. The display panel DP is configured to display a composite image in an area limited to the composite regular polygon display area in the current frame image.

[0072] In some embodiments, the memory M further stores computer executable instructions for controlling one or more processors P to determine a first field of view corresponding to the total display area; determine a second field of view corresponding to the second regular polygon display area; obtain a screen resolution corresponding to the first field of view; determine a current image resolution based on the first field of view, the second field of view, and the screen resolution; render an intermediate image based on the current image resolution and the second field of view; and send a data signal of the intermediate image to an integrated circuit connected to the display panel to display a composite image. In some embodiments, the second regular polygon display area is used as a reduced display area for determining the second field of view. The reduced image resolution is used as the current image resolution of the composite image in the current frame image. The intermediate image is rendered based on the current image resolution and the second field of view.

[0073] In some embodiments, the integrated circuit IC is configured to provide gate scanning signals and data signals to sub-pixels of a limited subset of the plurality of sub-pixels of the display panel DP. The composite image includes an intermediate image in a first area of ​​the composite regular polygon display area and a background image in a second area of ​​the composite regular polygon display area, the first area corresponding to the second regular polygon display area, and the second area does not overlap with the first area. The sub-pixels of the limited subset correspond to the composite regular polygon display area. Optionally, the integrated circuit IC is configured to provide the data signal of the intermediate image to the sub-pixels corresponding to the first area; and provide the background data signal to the sub-pixels corresponding to the second area.

[0074] In some embodiments, the integrated circuit IC is configured to provide gate scan signals and data signals limited to sub-pixels having a grayscale greater than a threshold value. Therefore, the limited subset of sub-pixels includes sub-pixels having a grayscale greater than the threshold value, and the limited subset of sub-pixels corresponds to a limited area of ​​the synthetic regular polygon display area.

[0075] In some embodiments, the memory M further stores computer executable instructions for controlling the one or more processors P to generate a virtual data signal including information about the size and position of the second virtual object and the current image resolution when rendering the intermediate image; and sending the virtual data signal to the integrated circuit together with the data signal of the intermediate image. The integrated circuit IC is configured to cache the virtual data signal in the integrated circuit for use in the next adjacent frame image.

[0076] In some embodiments, the first regular polygon display area is the minimum polygon display area capable of displaying the first virtual object in the previous adjacent frame image; the second regular polygon display area is the minimum polygon display area capable of displaying the second virtual object in the current frame image; and the composite regular polygon display area is the minimum polygon display area including the first regular polygon display area and the second regular polygon display area. Optionally, the first regular polygon display area and the second regular polygon display area overlap each other.

[0077] In some embodiments, the first regular polygon display area, the second regular polygon display area and the composite regular polygon display area are polygonal shapes of the same type. Optionally, the first regular polygon display area is a first rectangular display area; the second regular polygon display area is a second rectangular display area; and the composite regular polygon display area is a composite rectangular display area.

[0078] In some embodiments, the AR / VR image display device is a VR image display device. The display panel DP is configured to display the intermediate image in the reduced display area; and display the background image in an area outside the reduced display area of ​​the total display area.

[0079] In some embodiments, the integrated circuit IC is configured to provide gate scan signals and data signals limited to a limited subset of sub-pixels among a plurality of sub-pixels of the display panel. The limited subset of sub-pixels corresponds to a reduced display area. The integrated circuit IC is also configured to directly assign grayscale values ​​to sub-pixels in an area outside the reduced display area of ​​the entire display area, thereby displaying a background image.

[0080] In some embodiments, the reduced display area includes a high-resolution area and a low-resolution area. The limited subset of sub-pixels includes all sub-pixels corresponding to the high-resolution area and only sub-pixels corresponding to alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area. The integrated circuit IC is configured to provide gate scan signals and data signals limited to all sub-pixels corresponding to the high-resolution area and only sub-pixels corresponding to alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area.

[0081] In some embodiments, the memory M further stores computer executable instructions for controlling one or more processors P to obtain the coordinates of the gaze point; based on the coordinates of the gaze point, render a high-definition image for display in the sub-pixels corresponding to the high-resolution area; render a low-definition image for display in the alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area; process the high-definition image and the low-definition image using an image processing algorithm to generate a processed high-definition image and a processed low-definition image; and send the processed high-definition image and the processed low-definition image to the integrated circuit. The intermediate image includes the processed high-definition image and the processed low-definition image. Optionally, the image processing algorithm is one selected from the group consisting of an image enhancement algorithm and a local dimming algorithm.

[0082] In some embodiments, the VR image display device also includes a gaze tracker for tracking the user's gaze.

[0083] In some embodiments, the integrated circuit IC is further configured to stretch the processed low-definition image to generate a stretched low-definition image; and stitch the stretched low-definition image and the processed high-definition image together to generate a stitched image. The display panel DP is configured to display the stitched image.

[0084] In some embodiments, the memory M also stores computer executable instructions for controlling one or more processors P to generate a virtual data signal including information of a reduced size and position of a display area and a reduced image resolution when rendering an intermediate image; and sending the virtual data signal together with the data signal of the intermediate image to the integrated circuit IC.

[0085] In another aspect, the present disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions are executable by a processor to cause the processor to perform determining a first field of view corresponding to a total display area; determining a second field of view corresponding to a reduced display area; obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; and sending a data signal of the intermediate image to an integrated circuit connected to a display panel.

[0086] In some embodiments, computer-readable instructions may be executed by a processor to cause the processor to determine the size and position of a first virtual object to be displayed in a previous adjacent frame image; determine the size and position of a second virtual object to be displayed in a current frame image, wherein the previous adjacent frame image and the current frame image are two consecutive frame images; determine a first regular polygon display area in which the first virtual object in the previous adjacent frame image will be displayed, the first regular polygon display area being smaller than a total display area; determine a second regular polygon display area in which the second virtual object in the current frame image will be displayed, the second regular polygon display area being smaller than the total display area; and determine a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, is equal to or larger than the first regular polygon display area, and is equal to or larger than the second regular polygon display area.

[0087] In some embodiments, a second regular polygonal display area is used as a reduced display area for determining a second field of view; the reduced image resolution is used as a current image resolution of a composite image in a current frame image; and an intermediate image is rendered based on the current image resolution and the second field of view.

[0088] In some embodiments, the computer readable instructions may be executed by the processor to cause the processor to generate a virtual data signal including information of the size and position of the second virtual object and the current image resolution when rendering the intermediate image.

[0089] In some embodiments, the first regular polygon display area is the minimum polygon display area capable of displaying the first virtual object in the previous adjacent frame image; the second regular polygon display area is the minimum polygon display area capable of displaying the second virtual object in the current frame image; and the composite regular polygon display area is the minimum polygon display area including the first regular polygon display area and the second regular polygon display area. Optionally, the first regular polygon display area and the second regular polygon display area overlap each other. Optionally, the first regular polygon display area, the second regular polygon display area and the composite regular polygon display area are polygonal shapes of the same type. Optionally, the first regular polygon display area is a first rectangular display area; the second regular polygon display area is a second rectangular display area; and the composite regular polygon display area is a composite rectangular display area.

[0090] In some embodiments, the computer readable instructions are executable by the processor to cause the processor to perform: based on the coordinates of the gaze point, rendering a high-definition image for display in the sub-pixels corresponding to the high-resolution area; rendering a low-definition image for display in the alternating rows of sub-pixels in the sub-pixels corresponding to the low-resolution area; and processing the high-definition image and the low-definition image using an image processing algorithm to generate a processed high-definition image and a processed low-definition image. The intermediate image includes the processed high-definition image and the processed low-definition image. Optionally, the image processing algorithm is one selected from the group consisting of an image enhancement algorithm and a local dimming algorithm.

[0091] In some embodiments, the computer readable instructions are executable by a processor to cause the processor to generate a virtual data signal including a reduced size and position of a display area and a reduced image resolution when rendering an intermediate image.

[0092] For the purpose of illustration and description, the above description of the embodiments of the present invention has been given. It is not exhaustive, nor is it intended to limit the present invention to the precise form or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described to explain the principles of the present invention and its best mode practical application, so that those skilled in the art can understand the various embodiments of the present invention and the various modifications suitable for the specific use or implementation under consideration. The scope of the present invention is intended to be defined by the appended claims and their equivalents, wherein all terms are meant to have the broadest reasonable meaning unless otherwise stated. Therefore, the term "the invention" and the like do not necessarily limit the scope of the claims to a specific embodiment, and reference to the exemplary embodiments of the present invention does not mean a limitation of the present invention, and such limitation will not be inferred. The present invention is limited only by the spirit and scope of the appended claims. In addition, these claims may involve the use of "first", "second", etc., followed by a noun or element. These terms should be understood as nomenclature, and should not be interpreted as limiting the number of elements modified by these nomenclatures unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the present invention. It should be understood that those skilled in the art may make changes to the described embodiments without departing from the scope of the invention as defined by the appended claims. In addition, none of the elements and assemblies in this disclosure are intended to be dedicated to the public, regardless of whether the element or assembly is explicitly described in the appended claims.

Claims

1. An augmented reality / virtual reality (AR / VR) image display method, comprising: determining a first field of view corresponding to the total display area; determining a second field of view corresponding to the reduced display area; Obtaining a screen resolution corresponding to the first field of view; determining a reduced image resolution based on the first field of view, the second field of view, and the screen resolution; rendering an intermediate image based on the reduced image resolution and the second field of view; displaying the intermediate image in the reduced display area; The reduced display area includes a high-resolution area and a low-resolution area; and The data signal of the intermediate image is sent to an integrated circuit connected to a display panel.

2. The AR / VR image display method according to claim 1, wherein: The method is used for AR image display; The method further includes: Determine the size and position of a first virtual object to be displayed in a previous adjacent frame image; Determining a size and a position of a second virtual object to be displayed in a current frame image, wherein the previous adjacent frame image and the current frame image are two consecutive frame images; Determine a first regular polygon display area where the first virtual object in the previous adjacent frame image will be displayed, wherein the first regular polygon display area is smaller than the total display area; determining a second regular polygon display area where the second virtual object in the current frame image will be displayed, the second regular polygon display area being smaller than the total display area; determining a composite regular polygon display area including the first regular polygon display area and the second regular polygon display area, wherein the composite regular polygon display area is smaller than the total display area, equal to or larger than the first regular polygon display area, and equal to or larger than the second regular polygon display area; and In the current frame image, a synthesized image is displayed in an area limited to the synthesized regular polygon display area.

3. The AR / VR image display method according to claim 2, wherein: The second regular polygonal display area is used as the reduced display area to determine the second field of view; using the reduced image resolution as a current image resolution of the synthesized image in the current frame image; as well as The intermediate image is rendered based on the current image resolution and the second field of view.

4. The AR / VR image display method according to claim 3, wherein: Displaying the composite image in the current frame image includes providing gate scan signals and data signals restricted to a limited subset of sub-pixels among a plurality of sub-pixels of the display panel.

5. The AR / VR image display method according to claim 4, wherein: The composite image includes the intermediate image in a first area of ​​the composite regular polygon display area and a background image in a second area of ​​the composite regular polygon display area, the first area corresponds to the second regular polygon display area, and the second area does not overlap with the first area; The sub-pixels of the finite subset correspond to the synthetic regular polygon display area; providing the data signal of the intermediate image to sub-pixels corresponding to the first area; as well as A background data signal is provided to the sub-pixels corresponding to the second area.

6. The AR / VR image display method according to claim 4, wherein: The limited subset of sub-pixels includes sub-pixels with grayscale greater than a threshold; and The finite subset of sub-pixels corresponds to a finite area of ​​the composite regular polygon display area.

7. The AR / VR image display method according to any one of claims 3 to 6, wherein: Rendering the intermediate image includes generating a virtual data signal including information of the size and position of the second virtual object and the current image resolution; as well as The method further comprises sending the dummy data signal together with the data signal of the intermediate image to the integrated circuit; and The dummy data signal is buffered in the integrated circuit for use in a next adjacent frame image.

8. The AR / VR image display method according to any one of claims 2 to 6, wherein: The first regular polygon display area is a minimum polygon display area capable of displaying the first virtual object in the previous adjacent frame image; The second regular polygon display area is a minimum polygon display area of ​​the second virtual object that can be displayed in the current frame image; as well as The composite regular polygon display area is a minimum polygon display area including the first regular polygon display area and the second regular polygon display area.

9. The AR / VR image display method according to claim 8, wherein: The first regular polygon display area and the second regular polygon display area overlap each other.

10. The AR / VR image display method according to any one of claims 2 to 6, wherein: The first regular polygon display area, the second regular polygon display area, and the synthesized regular polygon display area have the same type of polygon shape.

11. The AR / VR image display method according to claim 10, wherein: The first regular polygon display area is a first rectangular display area; The second regular polygon display area is a second rectangular display area; and The synthetic regular polygon display area is a synthetic rectangular display area.

12. The AR / VR image display method according to claim 1, wherein: The method is used for VR image display; The method further comprises: A background image is displayed in an area outside the reduced display area of ​​the total display area.

13. The AR / VR image display method according to claim 12, wherein: Displaying the intermediate image includes providing gate scan signals and data signals restricted to a limited subset of sub-pixels of a plurality of sub-pixels of the display panel; The limited subset of sub-pixels corresponds to the reduced display area; Displaying the background image includes directly assigning, by the integrated circuit, grayscale values ​​to sub-pixels in the area outside of the reduced display area of ​​the total display area.

14. The AR / VR image display method according to claim 13, wherein: as well as The limited subset of sub-pixels includes all sub-pixels corresponding to the high-resolution region and sub-pixels corresponding only to alternating rows of sub-pixels among the sub-pixels corresponding to the low-resolution region.

15. The AR / VR image display method according to claim 14, further comprising: Get the coordinates of the gaze point; Based on the coordinates of the gaze point, rendering a high-definition image for display in sub-pixels corresponding to the high-resolution area; rendering a low-resolution image for display in said alternating rows of sub-pixels in said sub-pixels corresponding to said low-resolution region; Processing the high-definition image and the low-definition image using an image processing algorithm to generate a processed high-definition image and a processed low-definition image; as well as sending the processed high-definition image and the processed low-definition image to the integrated circuit; The intermediate image includes the processed high-definition image and the processed low-definition image.

16. The AR / VR image display method according to claim 15, further comprising: stretching, by the integrated circuit, the processed low-resolution image to generate a stretched low-resolution image; splicing the stretched low-resolution image with the processed high-resolution image by the integrated circuit to generate a spliced ​​image; and The stitched image is displayed.

17. The AR / VR image display method according to claim 15, wherein: The image processing algorithm is one selected from the group consisting of an image enhancement algorithm and a local dimming algorithm.

18. The AR / VR image display method according to any one of claims 12 to 17, wherein: Rendering the intermediate image includes generating a virtual data signal including information of the size and position of the reduced display area and the reduced image resolution; as well as The method further comprises sending the dummy data signal together with the data signal of the intermediate image to the integrated circuit.

19. An augmented reality / virtual reality (AR / VR) image display device, comprising: Display panel; an integrated circuit connected to the display panel; Memory; as well as one or more processors; wherein the memory and the one or more processors are connected to each other; and The memory stores computer executable instructions for controlling the one or more processors to: The processor executes the AR / VR image display method according to any one of claims 1 to 18.

20. A computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform the AR / VR image display method of any one of claims 1 to 18.

Images