Display method, display device, head-mounted display and storage medium

Abstract

The invention discloses a display method, a display device, a head-mounted display and a storage medium. The method is applied to a head-mounted display, the head-mounted display comprises a first display and a second display, and the method comprises the following steps: rasterizing a scene to be rendered to generate a rendering grid; based on the first pixel offset, rendering the rendering grid in a first target frame to generate a first display image, the first target frame being an odd number frame or an even number frame; based on a second pixel offset, rendering the rendering grid in the first target frame to generate a second display image, the second pixel offset being different from the first pixel offset; the first display displays a first display image, the second display displays a second display image, and an optical module of the first display adjusts an emergent angle of display light corresponding to the first display image to adapt to first pixel offset; and the optical module of the second display adjusts the emergent angle of the display light corresponding to the second display image to adapt to the second pixel offset, so that the effect of improving the resolution can be achieved.

Description

Technical Field

[0001] The present application belongs to the field of virtual reality technology, and in particular, relates to a display method, a display device, a head-mounted display, and a non-transitory computer-readable storage medium. Background Art

[0002] Currently, a head mounted display (HMD) processes images and projects the processed images to human eyes, thereby providing an immersive experience for users.

[0003] Although the screen resolution of head-mounted displays has been continuously improved to ensure the user experience, compared with the human eye's ability to perceive details, the display effect of head-mounted displays is still insufficient and the display effect is poor, which affects the user experience. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a display method, a display device, a head-mounted display and a non-transient computer-readable storage medium, which can improve the display effect and enhance the user experience.

[0005] The display method of the present application is applied to a head-mounted display, the head-mounted display includes a first display and a second display, and the method includes: rasterizing a scene to be rendered to generate a rendering grid; based on a first pixel offset, rendering the rendering grid in a first target frame to generate a first display image, the first target frame is an odd frame or an even frame; based on a second pixel offset, rendering the rendering grid in the first target frame to generate a second display image, the second pixel offset is different from the first pixel offset; displaying the first display image through the first display and displaying the second display image through the second display; wherein the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

[0006] In some embodiments, the optical module includes: a polarization modulator, configured to adjust the polarization angle of the display light of the display image, the polarization angle includes a first polarization angle and a second polarization angle; a refraction device, configured to refract the display light, the polarization angle and the refraction angle correspond to each other, wherein the refraction angle corresponding to the first polarization angle is greater than 0, and the refraction angle corresponding to the second polarization angle is 0, and the polarization angles corresponding to the first display image and the second display image are the first polarization angle; an optical path adjustment component, configured to converge the display light so that the emitted light is directed to a preset area.

[0007] In some embodiments, the rotation angle of the refraction device of the first display is determined based on the first pixel offset, and the offset angle of the exit direction of the display light corresponding to the first polarization angle of the first display relative to the preset direction is equal to the offset angle corresponding to the first pixel offset; the rotation angle of the refraction device of the second display is determined based on the second pixel offset, and the offset angle of the exit direction of the display light corresponding to the first polarization angle of the second display relative to the preset direction is equal to the offset angle corresponding to the second pixel offset.

[0008] In some embodiments, the polarization modulator, the refraction device, and the optical path adjustment component are arranged in sequence along the emission direction of the display light.

[0009] In some embodiments, the method further includes: rendering the rendering grid in a second target frame to generate a third display image and a fourth display image, the first target frame being one of an odd frame and an even frame, the second target frame being the other of an odd frame and an even frame, and the third display image and the fourth display image are not pixel-shifted; displaying the third display image through the first display and displaying the fourth display image through the second display, and the interval duration between the first target frame and the second target frame being less than the integration time of the human eye.

[0010] In some embodiments, the optical module does not change the emission angles of display lights of the third display image and the fourth display image.

[0011] In some embodiments, the offset distances of the first pixel offset and the second pixel offset are both half a pixel.

[0012] In some embodiments, an angle between the offset directions of the first pixel offset and the second pixel offset is 90 degrees.

[0013] The display device of the present application is applied to a head-mounted display, the head-mounted display includes a first display and a second display, and the device includes a generation module, a first rendering module, a second rendering module and a display module. The generation module is used to rasterize the scene to be rendered to generate a rendering grid; the first rendering module is used to render the rendering grid in a first target frame based on a first pixel offset to generate a first display image, the first target frame is an odd frame or an even frame; the second rendering module is used to render the rendering grid in the first target frame based on a second pixel offset to generate a second display image, the second pixel offset is different from the first pixel offset; the display module is used to display the first display image through the first display, and the second display displays the second display image; the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

[0014] The head-mounted display of the present application includes a first display, a second display, a memory and a processor; the memory stores a computer program, and when the processor executes the program, the display method described in any of the above embodiments is implemented.

[0015] The non-transitory computer-readable storage medium of the present application stores a computer program thereon, and when the computer program is executed by a processor, the display method as described in any of the above embodiments is implemented.

[0016] The display method, display device, head-mounted display, and non-transitory computer-readable storage medium provided by the embodiments of the present application rasterize a scene to be rendered to generate a rendering grid, render the rendering grid based on a first pixel offset in a first target frame to generate a first display image, and render the rendering grid based on a second pixel offset to generate a second display image. The first display image is displayed on a first display, and the second display image is displayed on a second display.

[0017] Since the display positions of the display images of the first target frames and non-first target frames are different in the first display (or the second display), based on the visual persistence characteristics of the human eye, multiple frames of display images with different display positions are superimposed on each other in the human eye to achieve an effect of improving the resolution in the corresponding deflection direction (that is, the first display corresponds to the deflection direction of the first pixel offset, and the second display corresponds to the deflection direction of the second pixel offset).

[0018] When viewing the first display image and the second display image, based on the inter-ocular blur suppression mechanism of the human eye, the human eye can suppress the situation in which the resolution improvement in the fused image is not large, that is, the first display image and the second display image will both suppress the area with a smaller resolution improvement in the other display image based on the area with a larger resolution improvement. The human eye can perceive more scene details, improve the clarity of the image obtained by the human eye, and achieve the effect of further improving the resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0020] Figure 1 It is an application scenario diagram of the display method provided in the embodiment of the present application;

[0021] Figure 2 It is a flowchart of a display method provided in an embodiment of the present application;

[0022] Figure 3 is a scene schematic diagram of the display method provided in an embodiment of the present application;

[0023] Figure 4 is a scene schematic diagram of the display method provided in an embodiment of the present application;

[0024] Figure 5 It is a flowchart of a display method provided in an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a scene of a display method provided in an embodiment of the present application;

[0026] Figure 7 is a scene schematic diagram of the display method provided in an embodiment of the present application;

[0027] Figure 8 is a scene schematic diagram of the display method provided in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of a module of a display device provided in an embodiment of the present application;

[0029] Fig.10 is a scene schematic diagram of the display method provided in an embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0032] For ease of understanding, the technical background and application scenarios of this application are first introduced below:

[0033] 1. Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR includes Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR).

[0034] Augmented Reality (AR): also known as augmented reality, is a technology that cleverly integrates virtual information with the real world. It widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and other technical means to simulate computer-generated text, images, three-dimensional models, music, video and other virtual information, and apply it to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0035] Virtual Reality (VR): also known as virtual reality or spiritual realm technology. Virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments in three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other high-tech technologies. With the help of the graphics processing unit (GPU) in the VR device, the image in the current scene is processed to produce a realistic three-dimensional visual, tactile, olfactory and other sensory experiences in the virtual world, so that people in the virtual world have an immersive feeling.

[0036] Mixed Reality (MR) technology refers to a new visualization environment created by merging the real and virtual worlds, in which physical and digital objects coexist and can interact with the real world in real time and obtain information in a timely manner.

[0037] 2. Vertex processing: The vertices of 3D graphics have a three-dimensional coordinate. Through linear algebra calculations, the coordinate data of each vertex in the three-dimensional space is converted and drawn into the two-dimensional space of the display. At the same time, the color of the vertex is calculated for subsequent pixel color interpolation. This operation is called vertex processing.

[0038] 3. Rasterization: The primitive information generated during vertex processing is input into the rasterization stage. Primitives are assembled first, and then the fragments corresponding to the screen pixels covered by the primitives are determined through triangle traversal. After the rasterization stage, the primitives will be divided into basic units of pixel size, which are called fragments. Fragments are more like data representations of pixels. The final pixels are generated by the information in the fragments, and then the fragments are colored by the fragment shader. When operating fragment by fragment, the fragments undergo a series of tests. The tested fragments will be converted into pixels and finally presented in the frame buffer area.

[0039] 4. Field Programmable Gate Array (FPGA) is a programmable logic device, which is a semiconductor chip composed of a series of programmable logic gates. In the technical solution provided in the embodiment of the present application, it is mainly used to send gate circuit signals to control the signal generator to change the state of the polarization modulator.

[0040] 5. Framebuffer: A video output device drives a video display device from a memory buffer containing complete frame data. The technical solution provided in the embodiment of the present application is mainly used to generate a display image after determining the required resolution as the output frame through Framebuffer resampling.

[0041] 6. Fresnel lens: A thin lens made of polyolefin material by injection molding. There are equidistant teeth on one side of the lens. Through these teeth, the light in the specified spectral range can be reflected or refracted. In VR devices, Fresnel lens is used as the main optical module of VR devices due to its thin thickness.

[0042] 7. Birefringent crystal. When a beam of light wave is projected onto the crystal interface, two refracted light beams are generally generated. This phenomenon is called birefringence. Due to the anisotropy of the crystal material, the dispersion angle of the two refracted light beams is related to the direction of the optical axis and the refractive index of the ordinary and extraordinary light of the crystal. The crystal that produces birefringence is called a birefringent crystal. The birefringent crystal mainly used in the technical solution provided in the embodiment of the present application is a quartz crystal.

[0043] In order to more vividly illustrate the various technical solutions in the embodiments of the present disclosure, the image processing method of the present application is introduced by taking virtual reality (VR) and head-mounted display as an example of VR head display used in VR. It can be understood that for XR technology, the principle of implementing the image processing method of the present application is basically similar and will not be repeated here. It should be emphasized that this is only an example and is not a specific limitation on the scope of application of the embodiments of the present disclosure.

[0044] See also Figure 1 , Figure 1 : is an application scenario diagram of an image processing method provided in an embodiment of the present application. The application scenario provided in the present application includes a head-mounted display 100 (such as a VR head-mounted display), and the head-mounted display 100 includes a body 10, a first display 20, and a second display 30; or, it can also be a first display 30 and a second display 20, which is not limited in the present application. Below, the first display 20 and the second display 30 are taken as examples for introduction.

[0045] In some embodiments, the head-mounted display 100 may be a video projection head-mounted display, a holographic head-mounted display, an AR head-mounted display, a VR head-mounted display, etc.

[0046] In some embodiments, the head-mounted display 100 may include at least one image acquisition device 40 (such as a visible light camera, an infrared camera, a structured light camera, etc.) for acquiring a two-dimensional or three-dimensional image of an actual scene.

[0047] In some embodiments, the head mounted display 100 further includes a processor 50, which may include a main processor (Microcontroller Unit, MCU) 51 and a graphics processor (Graphics Processing Unit, GPU) 52. The GPU processor 52 may perform vertex processing, rasterization, pixel shading, and Framebuffer resampling on the scene to be rendered, and generate a corresponding display image.

[0048] In some embodiments, the first display 20 and the second display 30 are used to display the display image generated after the GPU processor 52 samples and renders.

[0049] In some embodiments, the first display 20 and the second display 30 also include respective corresponding optical modules, and the display light of the display image enters the human eye after passing through the optical module. The optical module includes a polarization modulator, a refraction device and an optical path adjustment component, and the polarization modulator, the refraction device and the optical path adjustment component are arranged in sequence along the exit direction of the display light.

[0050] In some embodiments, the MCU 51 can control the change of the state of the fast polarization modulator in the optical module to change the polarization state of the display light.

[0051] In some embodiments, the optical path adjustment component may be a Fresnel lens.

[0052] In some embodiments, the optical path adjustment component can also be a folded optical path component, and the folded optical path component can be a Pancake optical component, that is, an optical component consisting of two 1 / 4 phase delay plates (Quarter-Wave Plate, QWP), a semi-transparent and semi-reflective mirror (Beam Splitter, BS), and a reflective polarizer (Reflective Polarizer, RP).

[0053] Based on the introduction of the above-mentioned related scenarios, the embodiment of the present application provides a display method, which is described in detail below:

[0054] See also Figure 2 , Figure 2 011, 012, 013, and 014. The head-mounted display is a VR device, and the method is implemented by step 011, step 012, step 013, and step 014. The head-mounted display is an example of a VR device, and the following is a detailed description.

[0055] Step 011: Rasterize the scene to be rendered to generate a rendering mesh.

[0056] Among them, the scene to be rendered can be a scene or image that can be displayed on the first display and / or the second display after rendering. The scene to be rendered can be an actual scene or a virtual scene, etc.; the rendering grid is a grid formed by fragments after the scene to be rendered is rasterized.

[0057] Specifically, a first visual sensor (i.e., an image acquisition device) can be set on the head mounted display, and the position and direction of the first visual sensor can simulate the user's eyes. According to the position and direction of the first visual sensor, the scene to be rendered presented to the user in the current scene when the user uses the VR device is determined, and the scene to be rendered is rasterized to generate a rendering grid.

[0058] For example, the virtual scene can be a three-dimensional (3D) scene or a two-dimensional scene. Taking the virtual scene as a 3D scene as an example, the scene to be rendered in the current scene is determined according to the first visual sensor set to simulate the user's eyes, and the view of the scene to be rendered is transformed through the GPU of the head-mounted display. The three-dimensional coordinates of the 3D scene graph are converted into two-dimensional coordinates, and then vertex processing and rasterization are performed to obtain a rendering mesh of the scene to be rendered.

[0059] like Figure 3 As shown, after rasterizing the rendered scene, we can get the following Figure 3 The rendering grid shown, one pixel can correspond to one or more grids.

[0060] Step 012: Based on the first pixel offset, render the rendering grid in a first target frame to generate a first display image, where the first target frame is an odd frame or an even frame.

[0061] The first pixel offset may be a value less than 1 such as 0.1, 0.3, 0.5, 0.6, 0.9, etc. For example, if the first pixel offset is 0.5, then the pixel is offset by 0.5 pixels during rendering.

[0062] Specifically, the display image of a plurality of consecutive frames may be numbered; or, the frame numbers of the display images of a plurality of consecutive frames may be obtained, and then, according to the parity of the numbers or the frame numbers, the current frame may be determined to be an odd frame or an even frame. For example, when the frame number of the current frame is an odd number, the frame may be determined to be an odd frame. The first target frame is an odd frame or an even frame. Taking the first target frame as an odd frame as an example, when the first target frame is an odd frame, the current grid to be rendered is sampled and rendered according to the image data information of the grid that is offset by the first pixel from the current grid to be rendered, so as to obtain the pixels corresponding to the current grid to be rendered, and the first display image is generated by rendering each grid to be rendered with the first pixel offset.

[0063] For example, see Figure 3 , a pixel includes 4 rendering grids (the rendering grids are arranged in a 2*2 manner), such as pixel a includes rendering grid (1, 1), rendering grid (1, 2), rendering grid (2, 1) and rendering grid (2, 2), the first target frame is an odd frame, and the first pixel offset is 0.5 pixels along the diagonal direction of the grid to the lower right (that is, 0.5 pixels are offset to the right in the horizontal direction and to the bottom in the vertical direction). After rasterizing the scene to be rendered, the following is obtained: Figure 3The 10*10 rendering grid shown in the figure is offset by 0.5 pixels, that is, offset by 1 grid during sampling and rendering. For example, when sampling and rendering the rendering grid (m, n) of the mth row and nth column, the image information used is the image information of the rendering grid (m+1, n+1) of the m+1th row and n+1th column. For example, in the first frame, when sampling and rendering the rendering grid (1, 1) of the 1st row and 1st column, the image information used is the image information of the rendering grid (2, 2) of the 2nd row and 2nd column. When sampling and rendering the rendering grid (1, 2) of the 1st row and 2nd column, the image information used is the image information of the rendering grid (2, 3) of the 2nd row and 3rd column. In this way, the corresponding offset and rendering are performed on each grid in turn to obtain each pixel, thereby generating the corresponding first display image, and each pixel of the first display image is offset by 0.5 pixels during rendering.

[0064] Step 013: Based on a second pixel offset, rendering the rendering grid in the first target frame to generate a second display image, the second pixel offset being different from the first pixel offset.

[0065] Among them, the second pixel offset can be a value less than 1 such as 0.1, 0.3, 0.5, 0.6, 0.9, etc., and the second pixel offset is different from the first pixel offset. For example, the first pixel offset is an offset of 0.1 pixels in the horizontal direction, and the second pixel offset is an offset of 0.2 pixels in the horizontal direction; or, the second pixel offset is an offset of 0.5 pixels in the vertical direction, etc.

[0066] Optionally, the offset distances of the first pixel offset and the second pixel offset are both half a pixel.

[0067] Optionally, an included angle between the offset directions of the first pixel offset and the second pixel offset is 90 degrees.

[0068] The offset distances of the first pixel offset and the second pixel offset are both 0.5 pixels, and the angle between the offset directions of the first pixel offset and the second pixel offset is 90 degrees (90°). For example, the first pixel offset may be in the vertical direction, offset upward by 0.5 pixels, and the second pixel offset may be in the horizontal direction, offset (leftward or rightward) by 0.5 pixels, etc.

[0069] Specifically, in the first target frame, the grid to be rendered currently can also be sampled and rendered based on the image data information of the grid that is offset by the second pixel from the grid to be rendered currently, so as to obtain the pixels corresponding to the grid to be rendered currently, that is, by rendering each grid to be rendered with the second pixel offset, so as to generate a second display image.

[0070] That is to say, in the first target frame, the rendering grid is rendered according to the first pixel offset to generate a first display image, and the rendering grid is rendered according to the second pixel offset to generate a second display image, wherein the first pixel offset and the second pixel offset are both 0.5 pixels, and the angle between the first pixel offset and the second pixel offset is 90°.

[0071] For example, see Figure 4 , assuming that the first target frame is an odd frame, the first pixel offset is an offset of 0.5 pixels with an angle α with the horizontal direction, and based on the first pixel offset, a corresponding first display image is generated, then the second pixel offset can be as follows Figure 4 As shown, a 0.5 pixel offset with an angle of α with the vertical direction generates a corresponding second display image based on the second pixel offset. The angle between the first pixel offset and the second pixel offset is 90°. In other words, when the first pixel offset is a 0.5 pixel offset with an angle of α with the horizontal direction, the second pixel offset can be a 0.5 pixel offset with an angle of α+90° or α-90° with the horizontal direction.

[0072] Step 014: Display the first display image on the first display, and display the second display image on the second display.

[0073] Specifically, the first display image is displayed by the first display of the head mounted display, and the second display image is displayed by the second display. Figure 1 The first display may be the first display on the left side of the head-mounted display, and the second display may be the second display on the right side of the head-mounted display, and the first display image may be displayed on the first display while the second display image is displayed; or the first display may be the first display on the right side of the head-mounted display, and the second display may be the second display on the left side of the head-mounted display, and the first display image may be displayed on the first display while the second display image is displayed.

[0074] Optionally, the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

[0075] Among them, the optical module can be used to modulate light (such as focusing or dispersing, etc.) so that the human eye can see the image.

[0076] Specifically, for the convenience of description, the first target frame is an odd frame as an example for explanation. Since the first display image is generated based on the first pixel offset rendering in the odd frame, when the first display displays the first display image, the optical module of the first display can adjust the emission angle of the display light corresponding to the first display image, so that the first display image can be imaged at the expected position (i.e., the position matching the offset direction and offset distance of the first pixel offset). Similarly, in the odd frame, based on the second pixel offset rendering, when the second display displays the second display image, the optical module of the second display can adjust the emission angle of the display light corresponding to the second display image, so that the second display image can be imaged at the expected position (i.e., the position matching the offset direction and offset distance of the second pixel offset).

[0077] The persistence of vision of the human eye refers to the characteristic that the inter-frame images within the integration time can be superimposed on the retina, because it takes a certain integration time for the image to be formed on the retina. That is, based on the persistence of vision of the human eye, the human eye obtains the superimposed images within the integration time. When the rendered scene is continuously sampled and rendered to generate multiple frames of display images, when the total display time of the continuous multiple frames of display images is less than or equal to the preset human eye integration time, the resolution perceived by the human eye will exceed the original resolution of the screen, thereby achieving the effect of improving the resolution.

[0078] For example, generally speaking, the daytime integration time of the human eye is 50 milliseconds (ms). If the frame rate of the display device is 90 Hz, then the interval between frames is 1 second (s) / 90 Hz = 0.01111s = 11.11ms. When 4 frames are superimposed, the superposition time is 4*11.11ms = 44.44ms, which is less than the daytime integration time of the human eye of 50ms. Inter-frame imaging can be superimposed on the retina during the integration time, so that the resolution perceived by the human eye is close to the original resolution of the screen, thereby achieving the effect of improving the resolution.

[0079] In the first target frame (odd frame or even frame), based on the first pixel offset, a first display image is generated, and the optical module of the first display will adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset. That is, in the first display, the display positions of the display images of each first target frame and the non-first target frame are different. Based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed in the human eye to achieve the effect of improving the resolution in the corresponding deflection direction (i.e., the deflection direction of the first pixel offset); similarly, in the first target frame (odd frame or even frame), based on the second pixel offset, a second display image is generated. That is, in the second display, the display positions of the display images of each first target frame and the non-first target frame are different. Based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed in the human eye to achieve the effect of improving the resolution in the corresponding deflection direction (i.e., the deflection direction of the second pixel offset).

[0080] The human eye has an interocular blur suppression mechanism (IOBS), which can be used in the human visual system to reduce or suppress blur between the two eyes. When viewing with both eyes, the images received by the two eyes may not be completely consistent due to reasons such as visual differences between the two eyes, differences in the refractive power of the two eyes, or differences in the distance of the two eyes to the same object in the field of view. In order to maintain clear and stable vision, the brain integrates information from the two eyes. During the integration process, IOBS allows the brain to prioritize information from the clearer or more dominant eye, while suppressing or reducing the impact of blurred or inconsistent information from the other eye. In other words, IOBS can reduce the impact of blurred images on visual quality and perception. In the case of binocular image formation, the clear part of the binocular will suppress the blurred part.

[0081] Therefore, since the first display image and the second display image have different deflection directions when reaching the human eye, based on the visual persistence characteristics of the human eye, the first display image and the second display image can achieve the effect of improving the resolution in their respective corresponding deflection directions; and when watching the first display image and the second display image, based on the inter-ocular blur suppression mechanism of the human eye, the human eye can suppress the situation where the resolution improvement in the fused image is not large, and the human eye can perceive more scene details, thereby improving the clarity of the image obtained by the human eye and achieving the effect of further improving the resolution.

[0082] In this way, by rasterizing the scene to be rendered to generate a rendering grid, in the first target frame, based on the first pixel offset, the rendering grid is rendered to generate a first display image, based on the second pixel offset, the rendering grid is rendered to generate a second display image, the first display image is displayed on the first display, and the second display image is displayed on the second display. Since the display positions of the display images of each first target frame and the non-first target frame are different in the first display (or the second display), based on the visual persistence characteristics of the human eye, the continuous multiple frames of display images with different display positions are superimposed in the human eye to achieve the effect of improving the resolution in the corresponding deflection direction (that is, the first display corresponds to the deflection direction of the first pixel offset, and the second display corresponds to the deflection direction of the second pixel offset). When watching the first display image and the second display image, based on the inter-eye blur suppression mechanism of the human eye, the human eye can suppress the situation where the resolution improvement is not large in the fused image, that is, the first display image and the second display image will suppress the corresponding area with a smaller resolution improvement in the other display image based on the area with a larger resolution improvement, so that the human eye can perceive more scene details, improve the clarity of the image obtained by the human eye, and achieve the effect of further improving the resolution.

[0083] See also Figure 5 In some embodiments, the display method further comprises:

[0084] Step 015: Rendering the rendering grid in the second target frame to generate a third display image and a fourth display image, the first target frame is one of the odd frame and the even frame, the second target frame is the other of the odd frame and the even frame, and the third display image and the fourth display image are not pixel-shifted;

[0085] Step 016: The third display image is displayed on the first display, and the fourth display image is displayed on the second display, and the interval between the first target frame and the second target frame is shorter than the integration time of the human eye.

[0086] The first target frame is one of an odd frame and an even frame, and the second target frame is the other of an odd frame and an even frame. For example, the first target frame is an odd frame and the second target frame is an even frame; or, the first target frame is an even frame and the second target frame is an odd frame, and this application does not limit this.

[0087] Specifically, when rendering the rendering grid in the second target frame, no pixel offset is performed to generate corresponding third display image and fourth display image, and then the third display image is displayed through the first display, and the fourth display image is displayed on the second display.

[0088] For example, taking the first target frame as an odd frame and the second target frame as an even frame as an example, in frame 0, the left eye display displays the third display image and the right eye display displays the fourth display image, and in frame 1, the left eye display displays the first display image and the right eye display displays the second display image.

[0089] In the same frame (taking the 0th frame as an example), the left eye display displays the first display image and the right eye display displays the second display image. The first display image and the second display image are fused in both eyes. Based on the inter-ocular blur suppression mechanism of the human eye, the human eye can perceive more scene details. Since the interval between the first target frame and the second target frame (taking the 0th frame and the 1st frame as an example) is less than the integration time of the human eye, the first display image and the third display image can be superimposed on the human eye through the left eye display, and the second display image and the fourth display image can be superimposed on the human eye through the right eye display to achieve resolution improvement. In other words, on the basis of monocular superimposed imaging, binocular imaging is superimposed to supplement the rendering offset in different directions, further increase the detail information that the human eye can perceive, and achieve the effect of further improving the resolution.

[0090] Optionally, the optical module does not change the emission angles of display lights of the third display image and the fourth display image.

[0091] Specifically, no pixel shift is performed when rendering the third display image and the fourth display image. Therefore, when displaying, the corresponding optical module does not change the emission angle of the display light of the third display image and the fourth display image to avoid misalignment of the display images.

[0092] By dividing multiple consecutive frames into a first target frame and a second target frame, in the first target frame, based on the corresponding pixel offset, the rendering grid is rendered to generate a corresponding display image (such as generating a first display image based on the first pixel offset, and generating a second display image based on the second pixel offset), and the corresponding image is displayed in the corresponding display (such as the first display displays the first display image, and the second display displays the second display image); in the second target frame, without pixel offset, the rendering grid is rendered to generate corresponding display images (such as generating a third display image and a fourth display image), and the corresponding images are displayed in the corresponding display (such as the first display displays the third display image, and the second display displays the fourth display image), wherein the optical module of the display performs corresponding modulation on the display light according to the pixel offset (such as in the first target frame, the optical module of the first display adjusts the emission angle of the display light to adapt to the first pixel offset, and the optical module of the second display adjusts the emission angle of the display light to adapt to the second pixel offset; in the second target frame, the optical modules of the first display and the second display do not change the emission angle of the display light).

[0093] See also Figure 6 In some embodiments, the optical module includes a polarization modulator, a refraction device, and an optical path adjustment component. The polarization modulator is configured to adjust the polarization angle of the display light of the display image, and the polarization angle includes a first polarization angle and a second polarization angle;

[0094] a refraction device configured to refract the display light, wherein the polarization angle corresponds to the refraction angle, wherein the refraction angle corresponding to the first polarization angle is greater than 0, the refraction angle corresponding to the second polarization angle is 0, and the polarization angles corresponding to the first display image and the second display image are the first polarization angle;

[0095] The optical path adjustment component is configured to converge the display light so that the emitted light is directed to a preset area.

[0096] Among them, the polarization modulator can be a light modulator that can change the polarization state of light according to a certain rule. The polarization angle includes a first polarization angle and a second polarization angle, the first polarization angle and the second polarization angle are different, the first polarization angle is 90°, and the second polarization angle is 0°. The polarization angle corresponding to the first display image and the second display image is the first polarization angle, and the polarization angle corresponding to the third display image and the fourth display image is the second polarization angle. That is, the display light of the first display image and the second display image, after passing through the polarization modulator, its polarization angle will be modulated to the first polarization angle, and the display light of the third display image and the fourth display image, after passing through the polarization modulator, its polarization angle will be modulated to the second polarization angle.

[0097] Among them, the refraction device can be a birefringent crystal, such as quartz crystal, calcite, etc. After the display light passes through the refraction device, it is refracted corresponding to the refraction angle to change the exit position of the display light. The polarization angle corresponds to the refraction angle. For example, after the display light of the first display image (or the second display image) passes through the polarization modulator, its polarization angle will be modulated to the first polarization angle, and the display light of the first polarization angle will be refracted at the first refraction angle after passing through the refraction device; after the display light of the third display image (or the fourth display image) passes through the polarization modulator, its polarization angle will be modulated to the second polarization angle, and the display light of the second polarization angle will be refracted correspondingly at the second refraction angle after passing through the refraction device. Therefore, in the first display, the display light of the first display image and the display light of the third display image are offset by half a pixel distance; in the second display, the display light of the second display image and the display light of the fourth display image are offset by half a pixel distance.

[0098] Optionally, the rotation angle of the refraction device of the first display is determined based on the first pixel offset, and the offset angle of the exit direction of the display light of the first display corresponding to the first polarization angle relative to the preset direction is equal to the offset angle corresponding to the first pixel offset; the rotation angle of the refraction device of the second display is determined based on the second pixel offset, and the offset angle of the exit direction of the display light of the second display corresponding to the first polarization angle relative to the preset direction is equal to the offset angle corresponding to the second pixel offset.

[0099] The preset direction may be the direction in which the display light is emitted after passing through the refraction device when the refraction device does not rotate.

[0100] Specifically, see Figure 7 , taking the refracting device as a circular quartz crystal as an example, in order to adapt the offset directions of the first display (corresponding to the first pixel offset) and the second display (corresponding to the second pixel offset), the refracting devices corresponding to the first display and the second display need to be rotated (around the central axis of the corresponding optical module) so that the first display and the second display match the corresponding pixel offsets. Therefore, the offset angle of the emission direction of the display light corresponding to the first polarization angle of the first display relative to the preset direction is equal to the offset angle of the first pixel offset (such as Figure 7 As shown, assuming that α is the same, the offset angle of the emission direction of the display light corresponding to the first polarization angle of the second display relative to the preset direction is the same as the offset angle corresponding to the second pixel offset (such as Figure 7 Therefore, in the first target frame, in the first display and the second display, there is an angle of 90° between the display light of the first display image and the display light of the second display image.

[0101] The optical path adjustment component is configured to converge the display light so that the emitted light is directed to a preset area, such as a Fresnel lens, a folded optical path component, etc. The display light is converged after passing through the optical path adjustment component and directed to a preset area (such as the area where the human eye is located) to form an image in the human eye.

[0102] Optionally, the polarization modulator, the refraction device and the optical path adjustment component are arranged in sequence along the emission direction of the display light.

[0103] Specifically, for the convenience of description, the first target frame is an odd frame, the second target frame is an even frame, the refraction device is a quartz crystal, and the optical path adjustment component is a Fresnel lens, and the description is based on the display of the first display.

[0104] In the first display, the rendering grid of the first display image is rendered with a first pixel offset, and the rendering grid of the third display image is rendered without a pixel offset. It is assumed that counting starts from the 0th frame, and the 0th frame is set to be rendered without a pixel offset (assuming that the sampling points without pixel offset are the original sampling points). After rendering, the third display image is obtained. The first frame will be rendered based on the image information of the first pixel offset (such as an offset of 0.5 pixels along the pixel diagonal line toward the lower right corner) to obtain the first display image. For example, assuming that the resolution of the first display is n*n, according to the frame signal of the main processor, normal rasterization and shading processes can be performed in even frames (second target frames). In odd frames (first target frames), due to the first pixel offset during rendering, the sampling position during rasterization is changed (relative to the original sampling point, there is a position along the pixel diagonal, diagonally offset by 0.5 pixels in the lower right corner). Therefore, shading is performed according to the new sampling position in the shading stage, and finally the first display image with a resolution of n*n is output to the first display through Framebuffer resampling, thereby sampling more details.

[0105] The optical path principle is described below by taking the display light of the first display image passing through the optical module of the first display as an example.

[0106] After passing through the quartz crystal, the P-polarized light propagates as the o-light, and after passing through the quartz crystal, the S-polarized light propagates as the e-light; after entering the polarization modulator, the P-polarized light is adjusted to 0° (0 phase) and becomes P-polarized light, and after adjusting to 90° (π phase), becomes S-polarized light; after entering the polarization modulator, the S-polarized light is adjusted to 0° (0 phase) and becomes S-polarized light, and after adjusting to 90° (π phase), becomes P-polarized light.

[0107] The display light passes through the polarization modulator and quartz crystal for light adjustment in sequence, and finally converges through the Fresnel lens to enter the human eye. P polarized light, after passing through the quartz crystal, propagates in the manner of o light; S polarized light, after passing through the quartz crystal, propagates in the manner of e light. The pixel projection images of o light and e light are offset accordingly after passing through the quartz crystal (i.e., an offset corresponding to the offset distance (0.5 pixel) of the first pixel offset).

[0108] Based on the above optical path principle, the odd frame is determined as the first target frame and the even frame is determined as the second target frame, and the signals are transmitted to the FPGA. The FPGA controls the signal generator to send the corresponding control voltage to change the modulation state of the polarization modulator. In the even frame, the polarization modulator is used to modulate the light into 0° linear polarized light so that it is emitted along the original path. In the odd frame, the polarization modulator is used to modulate the light into 90° linear polarized light so that it is emitted with a displacement of 0.5 pixels. After the emitted light passes through the quartz crystal, it passes through the Fresnel lens and forms an image in the human eye.

[0109] Then, the thickness d of the quartz crystal should be designed as:

[0110]

[0111] Where L is the width of a pixel on the first display, Φ is the dispersion angle of the e-light, and the solution is:

[0112]

[0113] Among them, θ is the angle between the incident light and the optical axis, no is the refractive index of o light, and ne is the refractive index of e light.

[0114] For example, see Figure 8 , the first target frame is an odd frame, the second target frame is an even frame, and the first display image and the third display image are displayed on the first display as an example. Assume that from the 0th frame (i.e. Figure 8 t=0) starts counting. When generating the third display image, the state of the polarization modulator is changed by controlling (for example, by controlling the FPGA to send a corresponding gate circuit signal to control the signal generator, the signal generator sends a control voltage according to the received signal, and the state of the polarization modulator is changed according to the control voltage), so that the third display image of the 0th frame enters the polarization modulator through the display light (P polarized light) emitted by the first display, and the polarization modulator modulates the second polarization angle (0°, i.e., 0 phase) to obtain P polarized light. After passing through the quartz crystal, the P polarized light propagates in the manner of o light and is emitted along the original path. In the 1st frame (i.e. Figure 8 When t=1), based on the first pixel offset (the offset size is 0.5 pixels, the angle is α), a first display image is generated, and then based on the polarization modulator, the display light (P polarized light) of the first display image is modulated at a first polarization angle (90°, i.e., π phase) to obtain S polarized light. After passing through the quartz crystal, the S polarized light propagates in the form of e light, and the e light of the first frame is offset by 0.5 pixels relative to the o light of the 0th frame; finally, the adjusted display light passes through the Fresnel lens and converges on the human eye to form an image.

[0115] It should be pointed out that quartz crystal has a birefringent property. When light passes through quartz crystal, refraction and optical path difference will change, which will affect the light propagation path and imaging effect. The change of optical path difference will cause a slight shift in the light propagation path when raster sampling is performed at different positions, which will affect the imaging of the sampling position after the pixel is shifted. Therefore, in order to obtain an accurate imaging effect, the offset size of the first pixel offset can also be determined according to the thickness of the quartz crystal in actual applications.

[0116] In addition, the optical module can also be a polarization modulator, a birefringent crystal and a folded optical path component, wherein the folded optical path component can be a pancake component, and the polarization modulator, birefringent crystal and pancake component are arranged in sequence along the emitting direction of the display light. The principles are similar and will not be elaborated here.

[0117] The light modulation principle and optical path principle of the second display are similar to those of the first display and will not be described in detail here.

[0118] In this way, the polarization angle of the display light of the display image is modulated by the polarization modulator of the optical module, the refraction device adjusts the refraction angle of the display light, and the optical path adjustment component converges the display light that has completed polarization and refraction, and the outgoing light can be directed to a preset area (such as the area where the human eye is located). Since the polarization angle corresponds to the refraction angle, and the polarization angle corresponding to the first display image and the second display image is the first polarization angle (such as 90°), the corresponding refraction angle is greater than 0, and the polarization angle corresponding to the third display image and the fourth display image is the second polarization angle (such as 0°), and the corresponding refraction angle is equal to 0, in the first display, the first display image corresponding to the first target frame and the third display image corresponding to the second target frame, the display light of the two can be emitted at different refraction angles, and the display positions of the display images of different frames are different, so that continuous multi-frame display images are superimposed within the visual retention time of the human eye, and the resolution improvement effect in the first offset direction can be achieved without adding a display screen. Similarly, in the second display, the resolution improvement effect in the second offset direction can be achieved.

[0119] Since the rotation angle of the refraction device of the first display is determined based on the first pixel offset, and the rotation angle of the refraction device of the second display is determined based on the second pixel offset, in the first target frame, there is an angle of 90° between the display light of the first display image and the offset direction of the second display image. When the first display image and the second display image are fused in the human eye, the resolution improvement effect can be achieved at their respective corresponding offset angles. Based on IOBS, the human eye can perceive more scene details, thereby achieving the effect of further improving the resolution.

[0120] According to the method described in the above embodiment, the present application embodiment also provides a display device 300 for executing the steps in the above display method. Fig. 9 , Fig. 9 3 is a schematic diagram of the structure of a display device 300 provided in an embodiment of the present application. The display device 300 is applied to a head-mounted display, and the head-mounted display includes a first display and a second display. The display device 300 may include a generation module 301, a first rendering module 302, a second rendering module 303 and a display module 304. The generation module 301 is used to rasterize the scene to be rendered to generate a rendering grid; the first rendering module 302 is used to render the rendering grid in a first target frame based on a first pixel offset to generate a first display image, and the first target frame is an odd frame or an even frame; the second rendering module 303 is used to render the rendering grid in the first target frame based on a second pixel offset to generate a second display image, and the second pixel offset is different from the first pixel offset; the display module 304 is used to display the first display image through the first display, and the second display displays the second display image; wherein the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

[0121] It should be noted that the specific details of each module unit in the above-mentioned display device 300 have been described in detail in the embodiment of the above-mentioned display method, and will not be repeated here.

[0122] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0123] In some embodiments, the display device 300 in the embodiment of the present application can be implemented in hardware, such as an electronic device, or a component in an electronic device, such as an integrated circuit or a chip; the display device 300 can also be implemented in software, such as as an application installed in an electronic device.

[0124] In some embodiments, see Figure 1 , Figure 11 is a schematic diagram of the structure of the head mounted display provided in the embodiment of the present application. The head mounted display 100 includes a processor 50, a memory 60, a first display 20 and a second display 30. The memory 60 stores a computer program 61 that can be run on the processor 50. When the program 61 is executed by the processor 50, each process of the embodiment of the above display method is implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0125] See also Figure 1 and Fig.10 , the GPU is used to sample and render the scene to be rendered (such as Fig.10 The first display and the second display are fused and formed in the human eye. The pixel shading processing and the Framebuffer resampling processing are continuously switched according to the frame signal, and the state of the polarization modulator is synchronously selected to improve the resolution.

[0126] The present application also provides a non-transitory computer-readable storage medium. Fig.11 The non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the various processes of the above-mentioned display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0127] The processor may be a processor in the electronic device in the above embodiment. The computer readable storage medium may be a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0128] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A display method, characterized in that: Applied to a head mounted display, the head mounted display comprises a first display and a second display, the method comprising: Rasterize the scene to be rendered to generate a rendering mesh; Based on the first pixel offset, rendering the rendering grid in a first target frame to generate a first display image, the first target frame being an odd frame or an even frame; Rendering the rendering grid at the first target frame to generate a second display image based on a second pixel offset, wherein the second pixel offset is different from the first pixel offset; Displaying the first display image through the first display and displaying the second display image through the second display; Among them, the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

2. The display method according to claim 1, characterized in that: The optical module comprises: A polarization modulator configured to adjust a polarization angle of a display light of a display image, wherein the polarization angle includes a first polarization angle and a second polarization angle; a refraction device configured to refract the display light, the polarization angle and the refraction angle corresponding to each other, wherein the refraction angle corresponding to the first polarization angle is greater than 0, the refraction angle corresponding to the second polarization angle is 0, and the polarization angles corresponding to the first display image and the second display image are the first polarization angle; The optical path adjustment component is configured to converge the display light so that the emitted light is directed to a preset area.

3. The display method according to claim 2, characterized in that: The rotation angle of the refraction device of the first display is determined based on the first pixel offset, and the offset angle of the emission direction of the display light of the first display corresponding to the first polarization angle relative to the preset direction is equal to the offset angle corresponding to the first pixel offset; The rotation angle of the refraction device of the second display is determined based on the second pixel offset, and the offset angle of the emission direction of the display light of the second display corresponding to the first polarization angle relative to the preset direction is equal to the offset angle corresponding to the second pixel offset.

4. The display method according to claim 2, characterized in that: The polarization modulator, the refraction device and the optical path adjustment component are arranged in sequence along the emitting direction of the display light.

5. The display method according to any one of claims 1 to 4, characterized in that: The method further comprises: Rendering the rendering grid in a second target frame to generate a third display image and a fourth display image, wherein the first target frame is one of an odd frame and an even frame, and the second target frame is the other of the odd frame and the even frame, and the third display image and the fourth display image are not pixel-shifted; The third display image is displayed through the first display, and the fourth display image is displayed through the second display, and the interval between the first target frame and the second target frame is shorter than the integration time of the human eye.

6. The display method according to claim 5, characterized in that: The optical module does not change the emission angles of the display lights of the third display image and the fourth display image.

7. The display method according to any one of claims 1 to 6, characterized in that: The offset distances of the first pixel offset and the second pixel offset are both half a pixel.

8. The display method according to any one of claims 1 to 7, characterized in that: An included angle between the offset directions of the first pixel offset and the second pixel offset is 90 degrees.

9. A display device, characterized in that: Applied to a head mounted display, the head mounted display comprises a first display and a second display, and the device comprises: A generation module, used to rasterize the scene to be rendered to generate a rendering mesh; A first rendering module, configured to render the rendering grid in a first target frame based on a first pixel offset to generate a first display image, wherein the first target frame is an odd frame or an even frame; a second rendering module, configured to render the rendering grid in the first target frame based on a second pixel offset to generate a second display image, wherein the second pixel offset is different from the first pixel offset; A display module, configured to display the first display image through the first display, and display the second display image through the second display; Among them, the optical module of the first display is used to adjust the emission angle of the display light corresponding to the first display image to adapt to the first pixel offset; the optical module of the second display is used to adjust the emission angle of the display light corresponding to the second display image to adapt to the second pixel offset.

10. A head mounted display, characterized in that: The device comprises a first display, a second display, a memory and a processor; the memory stores a computer program, and the processor implements the display method according to any one of claims 1 to 8 when executing the program.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the display method according to any one of claims 1 to 8 is implemented.

Images