Display method, display apparatus, head-mounted display and storage medium

Abstract

Disclosed in the present application are a display method, a display apparatus (300), a head-mounted display (100), and a storage medium. The method comprises: on the basis of a first pixel offset, rendering at a first target frame rendering grids to generate a first display image; on the basis of a second pixel offset, rendering at the first target frame the rendering grids to generate a second display image, the second pixel offset being different from the first pixel offset; and a first display (20) displaying the first display image, and a second display (30) displaying the second display image, the first display (20) adapting to the first pixel offset, and the second display (30) adapting to the second pixel offset.

Description

Display method, display device, head-mounted display, and storage medium

[0001] Priority information

[0002] This application claims priority to and the benefit of the filing date of the patent application with the China National Intellectual Property Office, with the patent application number of “2024110805335” submitted on August 6, 2024, and incorporates it herein in its entirety by reference. TECHNICAL FIELD

[0003] The present application belongs to the technical field of virtual reality, and particularly relates to a display method, a display device, a head-mounted display, and a non-transitory computer readable storage medium. BACKGROUND

[0004] At present, a head-mounted display (HMD) provides an immersive experience for a user by processing an image and projecting the processed image to the human eye.

[0005] Although the screen resolution of the head-mounted display is continuously improved to ensure the user experience, compared with the perception ability of the human eye to details, the display effect of the head-mounted display is still insufficient, the display effect is poor, and the user experience is affected. SUMMARY

[0006] The present application aims to at least solve one of the technical problems in the prior art. To this end, the present application proposes a display method, a display device, a head-mounted display, and a non-transitory computer readable storage medium, which can improve the display effect and improve the user experience.

[0007] To this end, one object of the present application is to propose a display method applied to a head-mounted display, the head-mounted display comprising a first display and a second display, the method comprising: rasterizing a to-be-rendered scene to generate a rendering grid; rendering the rendering grid based on a first pixel offset in a first target frame to generate a first display image, the first target frame being an odd frame or an even frame; rendering the rendering grid based on a second pixel offset in the first target frame to generate a second display image, the second pixel offset being different from the first pixel offset; displaying the first display image through the first display and displaying the second display image through the second display; wherein an optical module of the first display is configured to adjust an exit angle of a display light ray corresponding to the first display image to adapt to the first pixel offset; and an optical module of the second display is configured to adjust an exit angle of a display light ray corresponding to the second display image to adapt to the second pixel offset.

[0008] Still another object of the present application is to provide a display device applied to a head-mounted display including a first display and a second display, the device including a generating module, a first rendering module, a second rendering module and a display module. The generating module is configured to rasterize a scene to be rendered to generate a rendering grid; the first rendering module is configured to render the rendering grid based on a first pixel offset to generate a first display image in a first target frame, the first target frame being an odd frame or an even frame; the second rendering module is configured to render the rendering grid based on a second pixel offset to generate a second display image in the first target frame, the second pixel offset being different from the first pixel offset; and the display module is configured to display the first display image through the first display and the second display image through the second display; wherein an optical module of the first display is configured to adjust an exit angle of display light corresponding to the first display image to adapt to the first pixel offset; and an optical module of the second display is configured to adjust an exit angle of display light corresponding to the second display image to adapt to the second pixel offset.

[0009] Still another object of the present application is to provide a head-mounted display including a first display, a second display, a memory and a processor; the memory stores a computer program, and the processor executes the program to implement the display method of any of the above embodiments.

[0010] Still another object of the present application is to provide a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the display method of any of the above embodiments.

[0011] The display method, display device, head-mounted display and non-transitory computer readable storage medium provided by the embodiments of the present application generate a rendering grid by rasterizing a scene to be rendered, render the rendering grid based on a first pixel offset to generate a first display image in a first target frame, render the rendering grid based on a second pixel offset to generate a second display image in the first target frame, display the first display image on a first display and display the second display image on a second display.

[0012] Since the display positions of the display images of each first target frame and non-first target frame are different in the first display (or the second display), based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed and imaged on the human eye, achieving the effect of improving the resolution in the corresponding deflection direction (i.e. the deflection direction corresponding to the first pixel offset for the first display, and the deflection direction corresponding to the second pixel offset for the second display).

[0013] When viewing the first display image and the second display image, based on the interocular blur inhibition mechanism of the human eye, the human eye can inhibit the case that the resolution improvement amplitude is not large in the fused image, that is, the first display image and the second display image will inhibit the area with smaller resolution improvement in the corresponding another display image based on the area with larger resolution improvement, the human eye can perceive more scene details, improve the clarity of the image obtained by the human eye, and achieve the effect of further improving the resolution. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the references to the figures, in which:

[0015] FIG. 1 is an application scenario diagram of a display method according to an embodiment of the present application;

[0016] FIG. 2 is a flowchart of a display method according to an embodiment of the present application;

[0017] FIG. 3 is a scenario diagram of a display method according to an embodiment of the present application;

[0018] FIG. 4 is a scenario diagram of a display method according to an embodiment of the present application;

[0019] FIG. 5 is a flowchart of a display method according to an embodiment of the present application;

[0020] FIG. 6 is a scenario diagram of a display method according to an embodiment of the present application;

[0021] FIG. 7 is a scenario diagram of a display method according to an embodiment of the present application;

[0022] FIG. 8 is a scenario diagram of a display method according to an embodiment of the present application;

[0023] FIG. 9 is a module diagram of a display device according to an embodiment of the present application;

[0024] FIG. 10 is a scenario diagram of a display method according to an embodiment of the present application;

[0025] FIG. 11 is a connection state diagram of a non-volatile computer readable storage medium and a processor according to some embodiments of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0027] For the convenience of understanding, the technical background and application scenarios of the present application are introduced as follows:

[0028] 1. Extended Reality (XR) refers to combining reality and virtuality through a computer to create a virtual environment that can be interacted with by humans. XR includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

[0029] Augmented Reality (AR): AR is a technology that skillfully combines virtual information with the real world. It widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing, and other technical means to simulate and imitate virtual information such as text, images, three-dimensional models, music, and videos generated by computers, and applies them to the real world, with the two types of information complementing each other to achieve the "augmentation" of the real world.

[0030] Virtual Reality (VR): VR is also known as virtual reality or spiritual reality technology. Virtual reality technology includes computer, electronic information, and simulation technology. Its basic implementation is based on computer technology, using and integrating three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other latest developments in high-tech fields. With the help of a graphics processing unit (GPU) in a VR device, the current scene image is processed to create a virtual world with realistic three-dimensional vision, touch, smell, and other sensory experiences, making people in the virtual world feel as if they were there.

[0031] Mixed Reality (MR): MR refers to a new visual environment that combines the real world and the virtual world, in which physical and digital objects coexist and can interact with the real world in real time and obtain information in a timely manner.

[0032] 2. Vertex processing: The vertices of a 3D graph have a three-dimensional coordinate. Through linear algebra calculations, the coordinate data of each vertex in three-dimensional space is converted and drawn to the two-dimensional space of the display, and the color of the vertex is calculated for subsequent pixel color interpolation. This operation is called vertex processing.

[0033] 3、Rasterization: vertex processing generated primitive information is input to the rasterization stage, first primitive assembly, and then determine the corresponding pixel points of the screen covered by the primitive through triangle traversal, the primitive after the rasterization stage will be divided into a basic unit of pixel size, that is, called a fragment, the fragment is more like a pixel data representation, the final pixel is generated by the information in the fragment, and then the fragment is colored by the fragment shader, in the per-fragment operation, the fragment is processed through a series of tests, and the fragment after the test is converted into a pixel and finally displayed in the frame buffer area.

[0034] 4、Field Programmable Gate Array (FPGA) is a programmable logic device, which is a semiconductor chip composed of a series of programmable logic gates. In the technical scheme provided in the embodiments of the present application, it is mainly applied to the gate circuit signal control signal generator to change the state of the polarization modulator.

[0035] 5、Framebuffer: a video output device drives a video display device from a memory buffer containing complete frame data. In the technical scheme provided in the embodiments of the present application, it is mainly applied to generate a display image after determining the size of the resolution required as the output frame through framebuffer resampling.

[0036] 6、Fresnel lens: a thin sheet lens made of polyolefin material by injection molding. On one side of the lens, there are equidistant notches, which can reflect or refract light in a specified spectral range. In VR devices, Fresnel lenses are used as the main optical module due to their thin thickness.

[0037] 7、Birefringent crystal: when a beam of light is projected onto the crystal interface, two refracted beams are generally produced, which is called birefringence. Due to the anisotropy of the crystal material, the dispersion angle of the two refracted light beams is related to the direction of the optical axis and the refractive index of the ordinary and extraordinary light of the crystal. The crystal that produces birefringence is called birefringent crystal. The birefringent crystal mainly applied in the technical scheme provided in the embodiments of the present application is quartz crystal.

[0038] To make the technical solutions in the embodiments of the present disclosure more clear, the image processing method of the present application is introduced below taking a virtual reality (VR) and a head-mounted display as an example. It can be understood that the principle of implementing the image processing method of the present application is basically similar for XR technology, and will not be repeated here. It needs to be emphasized that this is just an example and is not a specific limitation on the applicable scope of the embodiments of the present disclosure.

[0039] Please refer to FIG. 1, which is an application scenario diagram of an image processing method provided by an embodiment of the present application. The application scenario provided by the present application includes a head-mounted display 100 (such as a VR head-mounted display), which includes a body 10, a first display 20 and a second display 30; or it can also be a first display 30 and a second display 20, which is not limited by the present application. In the following, the first display 20 and the second display 30 are taken as an example for introduction.

[0040] In some embodiments, the head-mounted display 100 can be a video projection head-mounted display, a holographic head-mounted display, an AR head-mounted display, a VR head-mounted display, etc.

[0041] In some embodiments, the head-mounted display 100 can include at least one image acquisition device 40 (such as a visible light camera, an infrared camera, a structured light camera, etc.) for acquiring two-dimensional or three-dimensional images of the actual scene.

[0042] In some embodiments, the head-mounted display 100 further includes a processor 50, which can include a microcontroller unit (MCU) 51 and a graphics processing unit (GPU) 52. The GPU processor 52 can generate corresponding display images after processing the scene to be rendered, such as vertex processing, rasterization, pixel shading and framebuffer resampling.

[0043] In some embodiments, the first display 20 and the second display 30 are used to display the display images generated after the GPU processor 52 is sampled and rendered.

[0044] In some embodiments, the first display 20 and the second display 30 further include their respective optical modules, and the display light of the display image enters the human eye after passing through the optical module. The optical module includes a polarization modulator, a refractive device and an optical path adjusting assembly, which are arranged in the order of the exit direction of the display light.

[0045] In some embodiments, the MCU 51 can control the change of the state of the fast polarization modulator in the optical module to change the polarization state of the display light.

[0046] In some embodiments, the optical path adjusting component can be a Fresnel lens.

[0047] In some embodiments, the optical path adjusting component can also be a folded optical path component, which can be a Pancake optical component, i.e., an optical component composed of two pieces of 1 / 4 phase delay plate (QWP), beam splitter (BS), and reflective polarizer (RP).

[0048] Based on the introduction of the above related scenarios, the embodiment of the present application provides a display method, and the display method will be described in detail as follows:

[0049] Please refer to FIG. 2, which is a flowchart of the display method provided by the embodiment of the present application. The display method provided by the embodiment of the present application is applied to a head-mounted display, which includes a first display and a second display. The display method is implemented by steps 011, 012, 013 and 014. Taking the head-mounted display as a VR device as an example, the following will be specifically described.

[0050] Step 011: rasterizing a to-be-rendered scene to generate a rendering grid.

[0051] The to-be-rendered scene can be a scene or an image that can be displayed on the first display and / or the second display after rendering. The to-be-rendered scene can be an actual scene or a virtual scene, etc. The rendering grid is a grid formed by a pixel after the rasterization of the to-be-rendered scene.

[0052] Specifically, a first visual sensor (i.e., an image acquisition device) can be arranged on the head-mounted display, and the position and direction of the first visual sensor can simulate the eyes of a user. According to the position and direction of the first visual sensor, the to-be-rendered scene in the current scene that is presented to the user in the case that the user uses the VR device is determined, and the to-be-rendered scene is rasterized to generate a rendering grid.

[0053] For example, the virtual scene can be a three-dimensional (3d) scene or a two-dimensional scene. Taking the virtual scene as a 3d scene as an example, according to the first visual sensor arranged to simulate the eyes of the user, the to-be-rendered scene in the current scene is determined. The GPU of the head-mounted display performs view transformation on the to-be-rendered scene, converts the three-dimensional coordinates of the 3d scene graph into two-dimensional coordinates, and then performs vertex processing and rasterization, so as to obtain the rendering grid of the to-be-rendered scene.

[0054] As shown in FIG. 3, after rasterizing the scene to be rendered, a rendering grid as shown in FIG. 3 can be obtained, and one pixel can correspond to one or more grids.

[0055] Step 012: based on the first pixel offset, rendering the rendering grid in the first target frame to generate a first display image, the first target frame being an odd frame or an even frame.

[0056] The first pixel offset can be a number less than 1, such as 0.1, 0.3, 0.5, 0.6, 0.9, etc. For example, when the first pixel offset is 0.5, the rendering is offset by 0.5 pixels.

[0057] Specifically, the display images of the continuous multiple frames can be numbered, or the frame sequence numbers of the display images of the continuous multiple frames are obtained, and the current frame is determined to be an odd frame or an even frame according to the parity of the number or the frame sequence number. For example, when the frame sequence number of the current frame is odd, the frame is determined to be an odd frame. The first target frame is an odd frame or an even frame. Taking the first target frame as an odd frame as an example, when the frame is odd, the image data information of the grid at a distance with a first pixel offset from the current rendering grid is obtained, and the current rendering grid is sampled and rendered according to the image data information, so as to obtain the pixel corresponding to the current rendering grid. Through the rendering of each rendering grid with a first pixel offset, the first display image is generated.

[0058] For example, please continue to refer to FIG. 3, taking a pixel including 4 rendering grids (the arrangement of the rendering grids is 2*2) as an example, such as pixel a including rendering grid (1, 1), rendering grid (1, 2), rendering grid (2, 1) and rendering grid (2, 2), the first target frame is an odd frame, the first pixel offset is 0.5 pixels in the diagonal direction of the grid (that is, 0.5 pixels in the horizontal direction and 0.5 pixels in the vertical direction), and the rasterization of the to-be-rendered scene is performed, and 10*10 rendering grids shown in FIG. 3 are obtained. When sampling and rendering, the offset is 0.5 pixels, that is, 1 grid, for example, when the rendering grid (m, n) in the mth row and the nth column is sampled and rendered, the image information used is the image information of the rendering grid (m+1, n+1) in the (m+1)th row and the (n+1)th column. For example, when the rendering grid (1, 1) in the 1st row and the 1st column is sampled and rendered in the 1st frame, the image information used is the image information of the rendering grid (2, 2) in the 2nd row and the 2nd column, and when the rendering grid (1, 2) in the 1st row and the 2nd column is sampled and rendered, the image information used is the image information of the rendering grid (2, 3) in the 2nd row and the 3rd column. In this way, the corresponding offset and rendering are sequentially performed on each grid to obtain each pixel, so as to generate the corresponding first display image, and each pixel of the first display image has an offset of 0.5 pixels during rendering.

[0059] Step 013: based on the second pixel offset, rendering the rendering grids in the first target frame to generate a second display image, the second pixel offset being different from the first pixel offset.

[0060] The second pixel offset can be a value less than 1, such as 0.1, 0.3, 0.5, 0.6, 0.9, and the second pixel offset is different from the first pixel offset, for example, the first pixel offset is an offset of 0.1 pixels in the horizontal direction, and the second pixel offset is an offset of 0.2 pixels in the horizontal direction; or the second pixel offset is an offset of 0.5 pixels in the vertical direction, and the like.

[0061] Optionally, the offset distances of the first pixel offset and the second pixel offset are both half a pixel.

[0062] Optionally, the included angle of the offset directions of the first pixel offset and the second pixel offset is 90 degrees.

[0063] The offset distances of the first pixel offset and the second pixel offset are both 0.5 pixels, and the included angle of the offset directions of the first pixel offset and the second pixel offset is 90 degrees (90°). For example, the first pixel offset can be an offset of 0.5 pixels in the vertical direction, upward, and the second pixel offset can be an offset of 0.5 pixels in the horizontal direction, left or right, and the like.

[0064] Specifically, at the first target frame, the current to-be-rendered grid can also be sampled and rendered according to the image data information of the grid with a second pixel offset distance from the current to-be-rendered grid, so as to obtain the pixel corresponding to the current to-be-rendered grid, that is, by performing second pixel offset rendering on each to-be-rendered grid, a second display image is generated.

[0065] That is to say, at the first target frame, the rendering grid is rendered according to the first pixel offset to generate a first display image, and the rendering grid is rendered according to the second pixel offset to generate a second display image, wherein the first pixel offset and the second pixel offset are both 0.5 pixels, and the included angle between the first pixel offset and the second pixel offset is 90°.

[0066] For example, referring to FIG. 4, assuming that the first target frame is an odd frame, the first pixel offset is an offset of 0.5 pixels with an angle of a with the horizontal direction, and based on the first pixel offset, a corresponding first display image is generated, then the second pixel offset can be an offset of 0.5 pixels with an angle of a with the vertical direction, as shown in FIG. 4, and based on the second pixel offset, a corresponding second display image is generated. The included angle between the first pixel offset and the second pixel offset is 90°. In other words, in the case where the first pixel offset is an offset of 0.5 pixels with an angle of a with the horizontal direction, the second pixel offset can be an offset of 0.5 pixels with an angle of a+90° or a-90° with the horizontal direction.

[0067] Step 014: displaying the first display image through the first display and displaying the second display image through the second display.

[0068] Specifically, the first display image is displayed through the first display of the head-mounted display, and the second display image is displayed through the second display of the head-mounted display. For example, referring again to FIG. 1, the first display can be the first display on the left side of the head-mounted display, and the second display is the second display on the right side of the head-mounted display, and the first display image is displayed through the first display and the second display image is displayed through the second display; or the first display can be the first display on the right side of the head-mounted display, and the second display is the second display on the left side of the head-mounted display, and the first display image is displayed through the first display and the second display image is displayed through the second display.

[0069] Optionally, the optical module of the first display is used to adjust the exit angle of the display light corresponding to the first display image to adapt to the first pixel offset; and the optical module of the second display is used to adjust the exit angle of the display light corresponding to the second display image to adapt to the second pixel offset.

[0070] The optical module can be used to modulate (such as focusing or dispersing) light so that the human eye can see the image.

[0071] Specifically, for the convenience of description, taking the first target frame as an odd frame as an example for description. Since the first display image is generated based on the first pixel offset rendering at the odd frame, when the first display displays the first display image, the optical module of the first display can adjust the exit angle of the display light corresponding to the first display image, so that the first display image can be imaged at the expected position (i.e., the position matched with the offset direction and offset distance of the first pixel offset). Similarly, when the second display image is generated based on the second pixel offset rendering at the odd frame, when the second display displays the second display image, the optical module of the second display can adjust the exit angle of the display light corresponding to the second display image, so that the second display image can be imaged at the expected position (i.e., the position matched with the offset direction and offset distance of the second pixel offset).

[0072] The persistence of vision characteristic of the human eye refers to the characteristic that the inter-frame imaging within the integration time can be superimposed on the retina due to the imaging of the image on the retina requiring a certain integration time. That is, based on the persistence of vision characteristic of the human eye, the human eye obtains the superimposed image within the integration time. When the scene to be rendered is continuously sampled and rendered to generate multiple display images, in the case that the total display time length of the continuous multiple display images is less than or equal to the preset human eye integration time, the resolution perceived by the human eye will exceed the original resolution of the screen, so that the effect of resolution improvement can be achieved.

[0073] For example, generally speaking, the daytime integration time of the human eye is 50 milliseconds (ms). And the frame rate of the display of the display device is 90 Hertz (HZ), so the interval between frames is 1 second (s) / 90 HZ=0.01111 s=11.11 ms. Therefore, when 4 frames are superimposed, the superimposition time is 4*11.11 ms=44.44 ms, which is less than the daytime integration time of the human eye 50 ms. The inter-frame imaging can be superimposed on the retina within the integration time, so that the resolution perceived by the human eye is close to the original resolution of the screen, thereby achieving the effect of resolution improvement.

[0074] In the first target frame (odd frame or even frame), a first display image is generated based on a first pixel offset, and an optical module of the first display adjusts an exit angle of display light corresponding to the first display image to adapt to the first pixel offset. That is, in the first display, the display positions of the display images of each first target frame and non-first target frame are different, and based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed and imaged in the human eye, achieving the effect of improving the resolution in the corresponding deflection direction (i.e., the deflection direction of the first pixel offset). Similarly, in the first target frame (odd frame or even frame), a second display image is generated based on a second pixel offset. That is, in the second display, the display positions of the display images of each first target frame and non-first target frame are different, and based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed and imaged in the human eye, achieving the effect of improving the resolution in the corresponding deflection direction (i.e., the deflection direction of the second pixel offset).

[0075] The human eye has an interocular blur suppression (IOBS) mechanism, which can be used in the human visual system to reduce or suppress blur between the two eyes. When viewing with both eyes, the images received by the two eyes can not be completely consistent due to reasons such as visual differences between the two eyes, refractive differences between the two eyes, or distance differences between the two eyes for the same object in the field of view. In order to maintain the clarity and stability of vision, the brain integrates information from both eyes, and in the integration process, IOBS allows the brain to preferentially process information from the clearer or more dominant eye while suppressing or reducing the influence of blurred or inconsistent information from the other eye. That is, IOBS can reduce the impact of blurred images on visual quality and perception, and in the case of binocular image fusion, the clear part of the binocular image suppresses the blurred part.

[0076] Therefore, since the first display image and the second display image have different deflection directions when they reach the human eye, based on the visual persistence characteristics of the human eye, the first display image and the second display image can achieve the effect of improving the resolution in their respective deflection directions. In the case of viewing the first display image and the second display image, based on the interocular blur suppression mechanism of the human eye, the human eye can suppress the case where the resolution is not greatly improved in the fused image, and the human eye can perceive more scene details, thereby improving the clarity of the image obtained by the human eye and achieving the effect of further improving the resolution.

[0077] Thus, by rasterizing the to-be-rendered scene to generate a rendering grid, at the first target frame, the rendering grid is rendered based on the first pixel offset to generate a first display image, and the rendering grid is rendered based on the second pixel offset to generate a second display image, the first display image is displayed on the first display, and the second display image is displayed on the second display. Since the display positions of the display images of each first target frame and non-first target frame are different in the first display (or the second display), based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed and imaged by the human eye, achieving the effect of improving the resolution in the corresponding deflection direction (i.e., the first display corresponds to the deflection direction at the first pixel offset, and the second display corresponds to the deflection direction at the second pixel offset). When watching the first display image and the second display image, based on the interocular blur suppression mechanism of the human eye, the human eye can suppress the case where the resolution improvement is not large in the fused image, that is, the first display image and the second display image will be suppressed based on the area with large resolution improvement in the corresponding another display image. The human eye can perceive more scene details, improve the clarity of the image obtained by the human eye, and achieve the effect of further improving the resolution.

[0078] Referring to FIG. 5, in some embodiments, the display method further includes:

[0079] Step 015: rendering the rendering grid at the second target frame to generate a third display image and a fourth display image, the first target frame being one of the odd-numbered frames and the even-numbered frames, the second target frame being the other of the odd-numbered frames and the even-numbered frames, and the third display image and the fourth display image not being subjected to pixel offset;

[0080] Step 016: displaying the third display image by the first display and displaying the fourth display image by the second display, and the interval time between the first target frame and the second target frame being less than the integration time of the human eye.

[0081] The first target frame is one of the odd-numbered frames and the even-numbered frames, and the second target frame is the other of the odd-numbered frames and the even-numbered frames. For example, the first target frame is an odd-numbered frame, and the second target frame is an even-numbered frame; or the first target frame is an even-numbered frame, and the second target frame is an odd-numbered frame, which is not limited in the present application.

[0082] Specifically, when the rendering grid is rendered at the second target frame, no pixel offset is performed to generate the corresponding third display image and fourth display image, and then the third display image is displayed by the first display and the fourth display image is displayed by the second display.

[0083] For example, taking the first target frame as an odd frame and the second target frame as an even frame as an example. At the 0th frame, the left eye display displays the third display image, and the right eye display displays the fourth display image. At the 1st frame, the left eye display displays the first display image, and the right eye display displays the second display image.

[0084] At the same frame (taking the 0th frame as an example), the left eye display displays the first display image, and the right eye display displays the second display image. The first display image and the second display image are fused in the eyes, and based on the interocular blur suppression mechanism of the human eye, the human eye can perceive more scene details. Since the interval time between the first target frame and the second target frame (taking the 0th frame and the 1st frame as an example) is less than the integration time of the human eye, the first display image and the third display image can be superimposed on the human eye through the left eye display, and the second display image and the fourth display image can be superimposed on the human eye through the right eye display, respectively, to achieve resolution enhancement. In other words, on the basis of monocular superimposed imaging, binocular superimposed imaging is added to supplement the rendering offset in different directions, further increase the details that the human eye can perceive, and achieve the effect of further improving the resolution.

[0085] Optionally, the optical module does not change the exit angle of the display light rays of the third display image and the fourth display image.

[0086] Specifically, the third display image and the fourth display image are not subjected to pixel offset during rendering, and therefore, the corresponding optical module does not change the exit angle of the display light rays of the third display image and the fourth display image during display, so as to avoid misplacement of the display images and the like.

[0087] By dividing the continuous multiple frames into the first target frame and the second target frame, at the first target frame, the rendering grid is rendered based on the corresponding pixel offset to generate the corresponding display image (such as generating the first display image based on the first pixel offset and generating the second display image based on the second pixel offset), and the corresponding display image is displayed in the corresponding display (such as the first display displaying the first display image and the second display displaying the second display image). At the second target frame, the rendering grid is rendered without pixel offset to generate the corresponding display image (such as generating the third display image and the fourth display image), and the corresponding display image is displayed in the corresponding display (such as the first display displaying the third display image and the second display displaying the fourth display image). The optical module of the display adjusts the display light rays according to the pixel offset (such as the exit angle of the display light rays of the optical module of the first display being adjusted to adapt to the first pixel offset and the exit angle of the display light rays of the optical module of the second display being adjusted to adapt to the second pixel offset at the first target frame, and the optical modules of the first display and the second display do not change the exit angle of the display light rays at the second target frame).

[0088] Referring to FIG. 6, in some embodiments, the optical module comprises a polarization modulator, a refractive device, and a light path adjusting assembly. The polarization modulator is configured to adjust a polarization angle of display light of a display image, the polarization angle comprising a first polarization angle and a second polarization angle;

[0089] The refractive device is configured to refract the display light, the polarization angle and a refraction angle corresponding, wherein the refraction angle corresponding to the first polarization angle is greater than 0, the refraction angle corresponding to the second polarization angle is 0, and the polarization angle corresponding to the first display image and the second display image is the first polarization angle;

[0090] The light path adjusting assembly is configured to converge the display light so that the emitted light is directed to a preset area.

[0091] The polarization modulator can be a light modulator capable of changing the polarization state of light according to a certain rule. The polarization angle comprises a first polarization angle and a second polarization angle, the first polarization angle and the second polarization angle being different, the first polarization angle being 90°, and the second polarization angle being 0°. The polarization angle corresponding to the first display image and the second display image is the first polarization angle, and the polarization angle corresponding to the third display image and the fourth display image is the second polarization angle. That is, the display light of the first display image and the second display image, after passing through the polarization modulator, its polarization angle will be modulated to the first polarization angle, and the display light of the third display image and the fourth display image, after passing through the polarization modulator, its polarization angle will be modulated to the second polarization angle.

[0092] The refractive device can be a birefringent crystal, such as quartz crystal, calcite, etc. After the display light passes through the refractive device, it is refracted corresponding to the refraction angle to change the emission position of the display light. The polarization angle corresponds to the refraction angle. For example, the display light of the first display image (or the second display image) passes through the polarization modulator, and its polarization angle is modulated to the first polarization angle. The display light of the first polarization angle passes through the refractive device and is refracted at the first refraction angle. The display light of the third display image (or the fourth display image) passes through the polarization modulator, and its polarization angle is modulated to the second polarization angle. The display light of the second polarization angle passes through the refractive device and is refracted at the second refraction angle. Thus, in the first display, the display light of the first display image and the display light of the third display image exist a half-pixel distance offset; in the second display, the display light of the second display image and the display light of the fourth display image exist a half-pixel distance offset.

[0093] Optionally, the rotation angle of the refractive device of the first display is determined based on the first pixel offset, and the offset angle of the exit direction of the display light corresponding to the first polarization angle of the first display relative to the preset direction is equal to the offset angle corresponding to the first pixel offset; the rotation angle of the refractive device of the second display is determined based on the second pixel offset, and the offset angle of the exit direction of the display light corresponding to the first polarization angle of the second display relative to the preset direction is equal to the offset angle corresponding to the second pixel offset.

[0094] The preset direction can be the exit direction of the display light after the display light passes through the refractive device when the refractive device does not rotate.

[0095] Specifically, referring to FIG. 7, taking a circular quartz crystal as an example of the refractive device, in order to adapt to the offset directions of the first display (corresponding to the first pixel offset) and the second display (corresponding to the second pixel offset), the refractive devices corresponding to the first display and the second display (around the central axis of the corresponding optical module) need to be rotated respectively to match the first display and the second display with the respective pixel offsets. Therefore, the offset angle of the exit direction of the display light corresponding to the first polarization angle of the first display relative to the preset direction is the same as the offset angle of the first pixel offset (as shown in FIG. 7, assuming α), and the offset angle of the exit direction of the display light corresponding to the first polarization angle of the second display relative to the preset direction is the same as the offset angle corresponding to the second pixel offset (as shown in FIG. 7, assuming α+90°). Therefore, in the first target frame, among the first display and the second display, the display light of the first display image and the display light of the second display image have a 90° angle.

[0096] The light path adjusting assembly is configured to converge the display light so that the exit light is directed to a preset area, such as a Fresnel lens, a folded light path assembly, etc. The display light is converged after passing through the light path adjusting assembly and is directed to a preset area (such as an area where a human eye is located) to form an image in the human eye.

[0097] Optionally, the polarization modulator, the refractive device, and the light path adjusting assembly are arranged in sequence along the exit direction of the display light.

[0098] Specifically, for the convenience of description, taking the first target frame as an odd frame, the second target frame as an even frame, the refractive device as a quartz crystal, and the light path adjusting assembly as a Fresnel lens as an example, the display of the first display is described.

[0099] In the first display, when the rendering grid of the first display image is rendered with a first pixel offset and the rendering grid of the third display image is rendered without a pixel offset, assuming that the count starts from the 0th frame, the 0th frame is set to be rendered without a pixel offset (assuming that the sampling point without a pixel offset is the original sampling point), and the third display image is obtained after rendering, then the 1st frame will be rendered based on the image information of the first pixel offset (such as 0.5 pixel offset along the pixel diagonal, obliquely to the lower right corner), to obtain the first display image. For example, assuming that the resolution of the first display is n*n, according to the frame signal of the main processor, the normal rasterization and shading process can be performed in the even frame (second target frame), and in the odd frame (first target frame), since the first pixel offset occurs during rendering, the sampling position during rasterization is changed (there is a 0.5 pixel offset along the pixel diagonal, obliquely to the lower right corner, relative to the original sampling point), therefore, the shading is performed according to the new sampling position in the shading stage, and finally the first display image with n*n resolution is output to the first display through framebuffer resampling, to realize sampling of more details.

[0100] The following describes the optical path principle by taking the display light of the first display image passing through the optical module of the first display as an example.

[0101] The P-polarized light propagates in the o light mode after passing through the quartz crystal, and the S-polarized light propagates in the e light mode after passing through the quartz crystal; the P-polarized light becomes P-polarized light after passing through the 0° (0 phase) adjustment of the polarization modulator, and becomes S-polarized light after passing through the 90° (π phase) adjustment; the S-polarized light becomes S-polarized light after passing through the 0° (0 phase) adjustment of the polarization modulator, and becomes P-polarized light after passing through the 90° (π phase) adjustment.

[0102] The display light passes through the polarization modulator and the quartz crystal in sequence to adjust the light, and finally converges through the Fresnel lens to enter the human eye. The P-polarized light propagates in the o light mode after passing through the quartz crystal, and the S-polarized light propagates in the e light mode after passing through the quartz crystal, so that the pixel projection images of the o light and the e light are offset by a corresponding offset distance (0.5 pixel offset) after passing through the quartz crystal.

[0103] Based on the above optical path principle, the odd frame is determined as the first target frame, and the even frame is determined as the second target frame, and is conveyed to the FPGA, and the corresponding control voltage is generated by the FPGA control signal generator to change the modulation state of the polarization modulator. In the even frame, the polarization modulator is used to modulate the light to 0° linearly polarized light, so that it is emitted according to the original path. In the odd frame, the polarization modulator is used to modulate the light to 90° linearly polarized light, so that it is offset by 0.5 pixels. After the emitted light passes through the quartz crystal, it is imaged in the human eye after passing through the Fresnel lens.

[0104] Therefore, the thickness d of the quartz crystal should be designed as:

[0105] Wherein, L is the width of one pixel on the first display, and Φ is the dispersion angle of e light, and the method is:

[0106] Wherein, θ is the included angle between the incident light and the optical axis, no is the refractive index of o light, and ne is the refractive index of e light.

[0107] For example, refer to FIG. 8, taking the first target frame as the odd frame and the second target frame as the even frame, and taking the first display image and the third display image as examples. Assuming that the counting starts from the 0th frame (i.e., t=0 in FIG. 8), in the case of generating the third display image, by controlling the state of the polarization modulator (for example, by controlling the FPGA to issue a corresponding gate circuit signal to control the signal generator, and the signal generator issues a control voltage according to the received signal to change the state of the polarization modulator), the third display image of the 0th frame is emitted by the first display light (P polarized light) into the polarization modulator, and the polarization modulator is modulated at the second polarization angle (0°, i.e., 0 phase) to obtain P polarized light. After the P polarized light passes through the quartz crystal, it propagates in the form of o light and is emitted according to the original path; in the 1st frame (i.e., t=1 in FIG. 8), based on the first pixel offset (the offset size is 0.5 pixels, and the angle is α), the first display image is generated, and then based on the polarization modulator, the display light (P polarized light) of the first display image is modulated at the first polarization angle (90°, i.e., π phase) to obtain S polarized light. After the S polarized light passes through the quartz crystal, it propagates in the form of e light, and the e light of the 1st frame is offset by 0.5 pixels from the o light of the 0th frame; finally, the adjusted display light is converged in the human eye by the Fresnel lens.

[0108] It should be noted that the quartz crystal has birefringence properties, when the light passes through the quartz crystal, refraction and optical path difference changes occur, which further affects the light propagation path and imaging effect, and the change in the optical path difference causes a slight shift in the propagation path of the light when grating sampling is performed at different positions, which further affects the imaging of the sampling position after the offset pixels. Thus, in order to obtain accurate imaging effect, the offset size of the first pixel offset can also be determined according to the thickness of the quartz crystal in actual application.

[0109] In addition, the optical module can also be a polarization modulator, a birefringent crystal and a folded light path assembly, wherein the folded light path assembly can be a pancake assembly, and the polarization modulator, the birefringent crystal and the pancake assembly are arranged in sequence along the exit direction of the display light. The principle is similar and will not be described here.

[0110] The light modulation principle and the light path principle of the second display are similar to those of the first display, and will not be described here.

[0111] In this way, the polarization angle of the display light of the display image is modulated by the polarization modulator of the optical module, the refraction angle of the display light is adjusted by the refraction device, and the display light that has completed polarization and refraction is converged by the light path adjustment assembly. The exit light can be shot to the preset area (such as the area where the human eye is located). Since the polarization angle and the refraction angle correspond, and the corresponding polarization angles of the first display image and the second display image are the first polarization angle (such as 90°), and the corresponding refraction angle is greater than 0, the corresponding polarization angles of the third display image and the fourth display image are the second polarization angle (such as 0°), and the corresponding refraction angle is equal to 0, in the first display, the display light of the first display image corresponding to the first target frame and the third display image corresponding to the second target frame can exit at different refraction angles. The display positions of the display images of different frames are different, which realizes the superposition of continuous multiple frames of display images within the visual persistence time of the human eye, and the resolution improvement effect in the first offset direction can be realized without increasing the display screen. Similarly, in the second display, the resolution improvement effect in the second offset direction can be realized.

[0112] Since the rotation angle of the refraction device of the first display is determined based on the first pixel offset, and the rotation angle of the refraction device of the second display is determined based on the second pixel offset, in the first target frame, the offset direction of the display light of the first display image and the second display image has a 90° angle, and when the first display image and the second display image are fused in the human eye, the resolution improvement effect can be realized based on the respective offset angles, and based on IOBS, the human eye can perceive more scene details, achieving the effect of further improving the resolution.

[0113] According to the method described in the above embodiments, the embodiments of the present application further provide a display device 300 for executing the steps in the display method described above. Please refer to FIG. 9, which is a structural schematic diagram of the display device 300 provided by the embodiments of the present application. The display device 300 is applied to a head-mounted display, and the head-mounted display includes a first display and a second display. The display device 300 can include a generating module 301, a first rendering module 302, a second rendering module 303 and a display module 304. The generating module 301 is configured to rasterize a scene to be rendered to generate a rendering grid; the first rendering module 302 is configured to render the rendering grid in a first target frame to generate a first display image based on a first pixel offset, the first target frame being an odd frame or an even frame; the second rendering module 303 is configured to render the rendering grid in the first target frame to generate a second display image based on a second pixel offset, the second pixel offset being different from the first pixel offset; and the display module 304 is configured to display the first display image through the first display and display the second display image through the second display; wherein an optical module of the first display is configured to adjust an exit angle of display light corresponding to the first display image to adapt to the first pixel offset; and an optical module of the second display is configured to adjust an exit angle of display light corresponding to the second display image to adapt to the second pixel offset.

[0114] It should be noted that the specific details of the modules in the display device 300 described above have been described in detail in the embodiments of the display method described above, and will not be described here.

[0115] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0116] In some embodiments, the display device 300 in the embodiments of the present application can be implemented in a hardware manner, such as an electronic device or a component in an electronic device, for example, an integrated circuit or a chip; the display device 300 can also be implemented in a software manner, such as an application installed in an electronic device.

[0117] In some embodiments, referring to FIG. 1, which is a structural schematic diagram of a head-mounted display provided in the embodiments, the head-mounted display 100 includes a processor 50, a memory 60, a first display 20 and a second display 30. The memory 60 stores a computer program 61 executable on the processor 50, which, when executed by the processor 50, implements the processes of the embodiments of the display method described above and achieves the same technical effects. To avoid repetition, details are not described herein.

[0118] Referring to FIG. 1 and FIG. 10, the GPU performs sampling rendering (such as vertex processing, rasterization, shading and framebuffer resampling in FIG. 10) on the scene to be rendered to generate a corresponding display image. The processor sends a first target frame control signal, and the MCU controls the signal generator to change the state of each polarization modulator according to the received signal to generate a control voltage. The display light is modulated to 0° or 90° linearly polarized light by the polarization modulator. The 0° or 90° linearly polarized light passing through the polarization modulator is converted into o light or e light by the birefringent crystal. The o light or e light passing through the birefringent crystal is superimposed and imaged on the human eye after passing through the optical module. The display images of the first display and the second display are fused and imaged in the human eye. The pixel shading and framebuffer resampling processes of the GPU are switched according to the frame signal, and the state of the polarization modulator is selected synchronously, so that the resolution is improved.

[0119] The embodiments of the present application also provide a non-transitory computer readable storage medium, referring to FIG. 11, the non-transitory computer readable storage medium stores a computer program, which, when executed by a processor, implements the processes of the embodiments of the display method described above and achieves the same technical effects. To avoid repetition, details are not described herein.

[0120] The processor can be the processor in the electronic device in the embodiments described above. The computer readable storage medium can be a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0121] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A display method characterized by comprising: The method is applied to a head-mounted display including a first display and a second display, and the method comprises: Rasterizing a to-be-rendered scene to generate a rendering grid; rendering the rendering grid in a first target frame based on a first pixel offset to generate a first display image, the first target frame being an odd frame or an even frame; rendering the rendering grid in the first target frame based on a second pixel offset to generate a second display image, the second pixel offset being different from the first pixel offset; displaying the first display image through the first display and displaying the second display image through the second display; wherein an optical module of the first display is configured to adjust an exit angle of display light corresponding to the first display image to adapt to the first pixel offset, and an optical module of the second display is configured to adjust an exit angle of display light corresponding to the second display image to adapt to the second pixel offset.

2. The display method according to claim 1, wherein The optical module comprises: a polarization modulator configured to adjust a polarization angle of display light of a display image, the polarization angle including a first polarization angle and a second polarization angle; a refractive device configured to refract the display light, the polarization angle and a refraction angle corresponding to each other, wherein a refraction angle corresponding to the first polarization angle is greater than 0, and a refraction angle corresponding to the second polarization angle is 0, and the first display image and the second display image correspond to the first polarization angle; a light path adjusting assembly configured to converge the display light so that the exit light is directed to a preset area.

3. The display method according to claim 2, wherein A rotation angle of the refractive device of the first display is determined based on the first pixel offset, and an offset angle of an exit direction of the display light corresponding to the first polarization angle of the first display relative to a preset direction is equal to an offset angle corresponding to the first pixel offset. A rotation angle of the refractive device of the second display is determined based on the second pixel offset, and an offset angle of an exit direction of the display light corresponding to the first polarization angle of the second display relative to the preset direction is equal to an offset angle corresponding to the second pixel offset.

4. The display method according to claim 2 or 3, wherein The polarization modulator, the refractive device, and the light path adjusting assembly are sequentially arranged along an exit direction of the display light.

5. The display method according to any one of claims 1 to 4, wherein The method further comprises: rendering the rendering grid in a second target frame to generate a third display image and a fourth display image, the first target frame being one of an odd frame and an even frame, the second target frame being the other of the odd frame and the even frame, and the third display image and the fourth display image not being subjected to pixel offset; displaying the third display image through the first display and displaying the fourth display image through the second display, and an interval time length between the first target frame and the second target frame being less than an integration time of a human eye.

6. The display method according to claim 5, wherein The optical module does not change an exit angle of display light of the third display image and the fourth display image.

7. The display method according to any one of claims 1 to 6, wherein The offset distance of the first pixel offset and the second pixel offset is half a pixel.

8. The display method according to any one of claims 1 to 7, wherein An included angle of offset directions of the first pixel offset and the second pixel offset is 90 degrees.

9. A display device comprising: Applied to a head-mounted display comprising a first display and a second display, the device comprises: a generating module configured to rasterize a scene to be rendered to generate a rendering grid; a first rendering module configured to render the rendering grid based on a first pixel offset to generate a first display image at a first target frame, the first target frame being an odd frame or an even frame; a second rendering module configured to render the rendering grid based on a second pixel offset to generate a second display image at the first target frame, the second pixel offset being different from the first pixel offset; a display module configured to display the first display image through the first display and the second display image through the second display; wherein an optical module of the first display is configured to adjust an exit angle of display light corresponding to the first display image to adapt to the first pixel offset, and an optical module of the second display is configured to adjust an exit angle of display light corresponding to the second display image to adapt to the second pixel offset.

10. A head-mounted display, comprising: The computer program is executed by the processor to implement the display method according to any one of claims 1-8. 11.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by the processor to implement the display method according to any one of claims 1-8.

Images