Display method, display device, display equipment and non-volatile computer readable storage medium
By adjusting the scene sampling, rendering and polarization angle of the virtual reality device, the problem of insufficient resolution of the virtual reality device is solved, and higher image clarity and dynamic effects are achieved, reducing hardware cost and complexity.
Patent Information
- Application Number
- CN202311689382.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-08
- Publication Date
- 2024-12-17
AI Technical Summary
Existing virtual reality devices have insufficient resolution, which affects the user experience, and methods to improve resolution such as multi-graphics and multi-display splicing have complexity, high cost and resolution limitations.
By sampling and rendering the current scene, a display image is generated, and the polarization angle of the display light is adjusted to match the sampling position, thereby achieving resolution superposition and improving image clarity.
While ensuring the frame rate remains unchanged, the resolution of virtual reality devices is improved, the shadowing is reduced, the dynamic effect is improved, and the hardware cost and system complexity are reduced.
Smart Images
Figure CN119155434A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual reality technology, and more specifically, to a display method, a display device, a display equipment, and a non-volatile computer-readable storage medium. Background Art
[0002] The resolution of the human retina is generally around 60 pixels per degree (PPD). When a user uses a virtual reality (VR) device, the closer the angular resolution of the VR device is to 60PPD, the closer the image clarity is to the resolution limit of the human eye, and the clearer the image seen by the human eye. However, the angular resolution of existing VR devices is generally around 20PPD, which is still very insufficient and affects the user experience of using VR devices. In the field of VR technology, improving the resolution of VR devices is still a problem that needs to be solved in this field. Summary of the invention
[0003] The embodiments of the present application provide a display method, a display device, a display equipment and a non-volatile computer-readable storage medium, which generate and display a display image by sampling and rendering different sampling positions of the current scene, and adjust the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene, so that the display image can achieve resolution superposition when superposition occurs in the human eye.
[0004] The display method of the embodiment of the present application includes: sampling and rendering the current scene to generate a display image, the sampling and rendering includes rasterization, and the sampling positions of multiple consecutive frames of the display image during rasterization are different; displaying the display image, and adjusting the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene, and the display positions of the display image corresponding to different polarization angles are different.
[0005] In some embodiments, the sampling rendering also includes pixel shading, and the sampling rendering of the current scene to generate a display image includes: determining the sampling position of the current scene according to a frame number corresponding to the current scene; rasterizing the current scene, and performing pixel shading on the current scene according to the sampling position of the current scene to generate the display image.
[0006] In some embodiments, determining the sampling position of the current scene based on the frame number corresponding to the current scene includes: when the frame number is an even number, determining the sampling position of the current scene as a first sampling position; when the frame number is an odd number, determining the sampling position of the current scene as a second sampling position, the first sampling position and the second sampling position are different.
[0007] In some embodiments, the continuous multiple frames of the display image include a first display image and a second display image, the first display image corresponds to the first sampling position, the second display image corresponds to the second sampling position, and in the diagonal direction of the pixel, there is a preset pixel displacement between the first sampling position and the second sampling position.
[0008] In some embodiments, the sampling rendering also includes vertex processing, and the sampling rendering of the current scene to generate a display image also includes: performing vertex processing on the current scene to determine a plurality of rendering areas to be sampled; rasterizing the current scene and performing pixel shading on the current scene according to the sampling position of the current scene to generate the display image, including: determining the rendering area to be sampled that matches each pixel of the current pixel array, the current pixel array being determined according to a preset sampling density; and performing pixel shading on pixels matching the rendering area to be sampled according to the sampling position of the current scene to generate the display image.
[0009] In some embodiments, it also includes: sending a current frame signal to a sampling rendering module and a display control module respectively to synchronize the sampling rendering module and the display control module, the sampling rendering module is configured to generate the display image, and the display control module is configured to control the polarization modulator to adjust the polarization angle of the display light by sending a control signal to the polarization modulator; when the sampling rendering module and the display control module are synchronized, the polarization angle matches the sampling position corresponding to the current scene.
[0010] In some embodiments, the total display time of the multiple consecutive frames of the displayed image is less than a preset human eye integration time.
[0011] The display device of the embodiment of the present application includes a generation module and a display module. The generation module is used to perform sampling and rendering on the current scene to generate a display image, the sampling and rendering includes rasterization, and the sampling positions of multiple consecutive frames of the display image are different when rasterization is performed; the display module is used to display the display image and adjust the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene.
[0012] The display device of an embodiment of the present application includes a processor, a memory; and a computer program, wherein the computer program is stored in the memory and executed by the processor, and the computer program includes instructions for executing the display method described in any of the above embodiments.
[0013] In some embodiments, the display device also includes a display configured to display the display image; a polarization modulator configured to adjust the polarization angle of the display light emitted by the display; a sampling rendering module configured to generate the display image; and a display control module configured to control the polarization modulator to adjust the polarization angle of the display light by sending a control signal to the polarization modulator.
[0014] In some embodiments, the display device also includes an optical module, which includes the polarization modulator, the birefringent crystal and the Fresnel lens, and the polarization modulator, the birefringent crystal and the Fresnel lens are arranged in sequence along the exit direction of the display light; or, the optical module includes the polarization modulator, the birefringent crystal and the folded optical path component, and the polarization modulator, the birefringent crystal and the folded optical path component are arranged in sequence along the exit direction of the display light.
[0015] The non-volatile computer-readable storage medium of the embodiment of the present application includes a computer program. When the computer program is executed by a processor, the processor executes the display method described in any of the above embodiments.
[0016] The display method, display device, display equipment and non-volatile computer-readable storage medium of the embodiment of the present application respectively set different sampling positions when rasterizing the current scene of multiple consecutive frames, so as to obtain display images under different scenes. Compared with the existing single-frame self-superposition method, the same scene is continuously sampled and rendered multiple times, and multiple display images under the same scene are superimposed to achieve resolution improvement, resulting in a small number of scenes contained in the continuous multiple-frame display images, and the frame rate of the continuous multiple-frame display images is lower than the frame rate of the scene data stream. In the present application, the scenes corresponding to the various display images are different, and the continuous multiple-frame display images contain a large number of scenes, which are basically the same as the frame rate generated by the scene data stream, thereby ensuring that the frame rate remains unchanged.
[0017] Under the premise of ensuring that the frame rate remains unchanged, the display images corresponding to different sampling positions of multiple consecutive frames are displayed using the polarization angles corresponding to the sampling positions, so that the display positions of the multiple consecutive frames of display images are different. Based on the visual persistence characteristics of the human eye, the multiple consecutive frames of display images with different display positions are superimposed on each other in the human eye to achieve improved resolution.
[0018] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0020] Figure 1 It is a schematic diagram of an application scenario of a display method of certain embodiments of the present application;
[0021] Figure 2 is a schematic flow chart of a display method of certain embodiments of the present application;
[0022] Figure 3 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0023] Figure 4 is a schematic flow chart of a display method of certain embodiments of the present application;
[0024] Figure 5 is a schematic flow chart of a display method of certain embodiments of the present application;
[0025] Figure 6 is a schematic flow chart of a display method of certain embodiments of the present application;
[0026] Figure 7 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0027] Figure 8 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0028] Fig. 9 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0029] Fig.10 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0030] Fig.11 It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0031] Fig.12 is a schematic flow chart of a display method of certain embodiments of the present application;
[0032] Fig.13 is a schematic diagram of a module of a display device according to some embodiments of the present application;
[0033] Fig.14It is a scene schematic diagram of a display method of certain embodiments of the present application;
[0034] Fig.15 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION
[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.
[0036] To facilitate understanding of this application, the terms appearing in this application are explained below:
[0037] 1. Virtual reality technology (VR), also known as virtual reality or spiritual technology. Virtual reality technology encompasses computers, electronic information, and simulation technology. Its basic implementation method is based on computer technology, utilizing and integrating the latest developments in a variety of high technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. The graphics processor (GPU) in the VR device processes the image in the current scene to produce a realistic three-dimensional visual, tactile, olfactory and other sensory experiences in the virtual world, so that people in the virtual world have an immersive feeling. The technical solution provided in the embodiment of the present application is mainly used to enhance the resolution of VR devices without affecting the frame rate.
[0038] 2. Vertex processing: The vertices of 3D graphics have a three-dimensional coordinate. Through linear algebra calculations, the coordinate data of each vertex in the three-dimensional space is converted and drawn into the two-dimensional space of the display. At the same time, the color of the vertex is calculated for subsequent pixel color interpolation. This operation is called vertex processing.
[0039] 3. Rasterization: The primitive information generated during vertex processing is input into the rasterization stage. Primitives are assembled first, and then the fragments corresponding to the screen pixels covered by the primitives are determined through triangle traversal. After the rasterization stage, the primitives will be divided into basic units of pixel size, which are called fragments. Fragments are more like data representations of pixels. The final pixels are generated by the information in the fragments, and then the fragments are colored by the fragment shader. When operating fragment by fragment, the fragments undergo a series of test processes. The tested fragments will be converted into pixels and finally presented in the frame buffer area.
[0040] 4. Field Programmable Gate Array (FPGA) is a programmable logic device, which is a semiconductor chip composed of a series of programmable logic gates. In the technical solution provided in the embodiment of the present application, it is mainly used to send gate circuit signals to control the signal generator to change the state of the polarization modulator.
[0041] 5. Framebuffer: A video output device drives a video display device from a memory buffer containing complete frame data. The technical solution provided in the embodiment of the present application is mainly used to generate a display image after determining the required resolution as the output frame through Framebuffer resampling.
[0042] 6. Fresnel lens: A thin lens made of polyolefin material by injection molding. There are equidistant teeth on one side of the lens. Through these teeth, the light in the specified spectral range can be reflected or refracted. In VR devices, Fresnel lens is used as the main optical module of VR devices due to its thin thickness.
[0043] 7. Birefringent crystal. When a beam of light wave is projected onto the crystal interface, two refracted light beams are generally generated. This phenomenon is called birefringence. Due to the anisotropy of the crystal material, the dispersion angle of the two refracted light beams is related to the direction of the optical axis and the refractive index of the ordinary and extraordinary light of the crystal. The crystal that produces birefringence is called a birefringent crystal. The birefringent crystal mainly used in the technical solution provided in the embodiment of the present application is a quartz crystal.
[0044] The resolution of the human retina is about 60 pixels per degree (PPD). It is generally believed that the closer the angular resolution of a VR device is to 60PPD, the closer the image clarity is to the resolution limit of the human eye, and the clearer the image is. When the resolution of a VR device is insufficient, the human eye will directly see the pixels of the display, resulting in a screen door effect. The angular resolution of existing VR devices is generally around 20PPD, so the resolution of VR devices is still significantly insufficient.
[0045] Currently, there is a method of increasing resolution by splicing multiple graphics cards and multiple displays to project different sampled images for superposition. However, the system structure with multiple graphics cards and multiple displays is complex, and it takes up a large physical space, which makes the VR device too heavy. In addition, the hardware cost is high, the system adjustability is poor, and the displayed image resolution is still limited due to the limitations of transmission signal bandwidth and processing performance.
[0046] In order to solve the above technical problems, an embodiment of the present application provides a display method.
[0047] The following first introduces the application scenarios of the technical solution of this application. Figure 1 , which is a schematic diagram of an application scenario of a display method provided in an embodiment of the present application, wherein the application scenario involves a display device 100 .
[0048] Figure 1 One display device 100 is shown by way of example, but other numbers of display devices 100 may actually be included, and this is not limited in the embodiments of the present application.
[0049] In one embodiment, the display device 100 includes a body 10, a display 20, a processor 30, and a memory 40. The display 20 is disposed inside the body 10 and can display images.
[0050] In one embodiment, the processor 30 includes a main processor 31 and a graphics processing unit (GPU) 32. The GPU processor 32 performs sampling and rendering on the current scene to generate a scene image.
[0051] In one embodiment, the display 20 is used to display the scene image generated after sampling and rendering by the GPU processor 30.
[0052] In one embodiment, the display device 100 also includes a display control module 60, and the GPU processor 30 includes a sampling rendering module 50. A frame signal is sent through the main processor 31 to synchronize the sampling rendering module 50 and the display control module 60 so that the polarization angles of the displayed image and the light modulated by the polarization modulator can correspond.
[0053] In one embodiment, the display device 100 includes an optical module, and the display light of the display image of the display 20 enters the human eye after passing through the optical module. The optical module includes a polarization modulator, a birefringent crystal and a Fresnel lens, and the polarization modulator, the birefringent crystal and the Fresnel lens are arranged in sequence along the emitting direction of the display light.
[0054] In one embodiment, the optical module also includes a polarization modulator, a birefringent crystal and a folded optical path component. The polarization modulator, the birefringent crystal and the folded optical path component are arranged in sequence along the exit direction of the display light. The folded optical path component can be a Pancake optical component, that is, an optical component including two 1 / 4 phase delay plates (Quarter-Wave Plate, QWP), a semi-transparent and semi-reflective mirror (Beam Splitter, BS), and a reflective polarizer (Reflective Polarizer, RP).
[0055] The display method of this application will be described in detail below:
[0056] See also Figure 1 and Figure 2 , an embodiment of the present application provides a display method, the display method comprising:
[0057] Step 011: Sample and render the current scene to generate a display image. The sample and render include rasterization. The sampling positions of consecutive frames of display images are different during rasterization.
[0058] The current scene is a frame of multiple frames of scenes included in the scene data stream, and the scene data stream is generated by a virtual three-dimensional (3D) scene or a two-dimensional scene built by an application.
[0059] Specifically, taking the current scene as a 3D scene as an example, the GPU performs a view transformation on the current scene, converts the three-dimensional coordinates of the 3D scene graph into two-dimensional coordinates, and converts one or more vertices into completely different basic graphics (or primitives), thereby generating many more vertices than the original. Then, the primitives are assembled through rasterization, and then the fragments corresponding to the screen pixels covered by the primitives are determined through triangle traversal. Finally, the fragments are colored and tested according to different sampling positions to generate the display image of the current scene.
[0060] For example, see Figure 3, perform view transformation on the current scene in the GPU, convert the three-dimensional coordinates of the 3D scene graph into two-dimensional coordinates, and determine multiple areas to be rendered in the current scene. Determine the matching pixels of the area to be rendered (such as the triangular area in the figure) as 2*2 through rasterization. In the first frame, sample and render the area to be rendered to obtain the corresponding display image of the first frame. In the second frame, sample and render the area with a distance offset by half a pixel to the lower right corner relative to the diagonal position of the first frame to obtain the corresponding display image of the second frame. Determine 2*2 pixels as the output frame through Framebuffer resampling to generate the display image of the current scene.
[0061] Optionally, the total display time of the continuous multiple frames of displayed images is less than a preset human eye integration time.
[0062] Specifically, it takes a certain integration time for an image to be formed on the retina, during which the inter-frame images can be superimposed on the retina. At this time, due to the visual persistence characteristics of the human eye, the human eye obtains the superimposed images within the integration time. When the current scene is continuously sampled and rendered to generate multiple frames of display images, if the total display time of the continuous multiple frames of display images is less than or equal to the preset human eye integration time, the resolution perceived by the human eye will exceed the original resolution of the screen, thereby achieving the effect of improving the resolution.
[0063] For example, generally speaking, the daytime integration time of the human eye is 50 milliseconds (ms). If the frame rate of the display device is 90 Hz, then the interval between frames is 1 second (s) / 90 Hz = 0.01111s = 11.11ms. When 4 frames are superimposed, the superposition time is 4*11.11ms = 44.44ms, which is less than the daytime integration time of the human eye of 50ms. Inter-frame imaging can be superimposed on the retina during the integration time, so that the resolution perceived by the human eye is close to the original resolution of the screen, thereby achieving the effect of improving the resolution.
[0064] Step 012: Display the display image, and adjust the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene. Different polarization angles correspond to different display positions of the display image.
[0065] Specifically, the generated display image is displayed by the display, and for each frame of the display image displayed by the display, the polarization angle of the display light emitted is adjusted, and the display light of different display images generated corresponding to different sampling positions is converted into different polarization states, and the polarization state corresponds to the sampling position corresponding to the current scene.
[0066] In this way, by acquiring the current scene in the scene data stream and using the corresponding sampling position for sampling and rendering, a display image corresponding to the current scene is generated. When multiple consecutive frames of the current scene are rasterized, different sampling positions are set respectively, so as to obtain multiple consecutive frames of display images corresponding to the multiple consecutive frames of the current scene.
[0067] Compared with the existing single-frame self-superposition method, the same scene is continuously sampled and rendered multiple times, and multiple display images of the same scene are superimposed to achieve resolution improvement, resulting in the continuous multi-frame display images containing fewer scenes, and the frame rate of the continuous multi-frame display images is lower than the frame rate of the scene data stream. The scenes corresponding to the various display images of the present application are different, and the continuous multi-frame display images contain a large number of scenes, which is basically the same as the frame rate generated by the scene data stream, thereby ensuring that the frame rate remains unchanged.
[0068] Under the premise of ensuring that the frame rate remains unchanged, the display images corresponding to different sampling positions of multiple consecutive frames are displayed using the polarization angles corresponding to the sampling positions, so that the display positions of the multiple consecutive frames of display images are different. Based on the visual persistence characteristics of the human eye, the multiple consecutive frames of display images with different display positions are superimposed on each other in the human eye to achieve improved resolution.
[0069] See also Figure 4 In some embodiments, the display method further includes:
[0070] Step 013: sending the current frame signal to the sampling rendering module and the display control module respectively to synchronize the sampling rendering module and the display control module, wherein the sampling rendering module is configured to generate a display image, and the display control module is configured to control the polarization modulator to adjust the polarization angle of the display light by sending a control signal to the polarization modulator;
[0071] Step 014: When the sampling rendering module and the display control module are synchronized, the polarization angle matches the sampling position corresponding to the current scene.
[0072] Among them, the polarization modulator is an optical modulator that changes the polarization state of light according to a certain rule.
[0073] Specifically, by sending the current frame signal to the sampling and rendering module and the display control module respectively, the sampling and rendering module and the display control module are synchronized so that the display light of the display image and the state of the polarization modulator can correspond. After receiving the current frame signal, the sampling and rendering module performs sampling and rendering on the current scene, such as through sampling and rendering processes such as vertex processing, rasterization, pixel shading and Framebuffer resampling to generate a display image. After receiving the current frame signal, the display control module sends a control signal to the polarization modulator to control the polarization modulator to adjust the polarization angle of the display light. For example, after receiving the current frame signal, the display control module sends a corresponding gate circuit signal to control the signal generator through the FPGA. The signal generator sends a control voltage according to the received signal to control the change of the state of the polarization modulator to achieve control of the polarization modulator to adjust the display light of the display image.
[0074] See also Figure 5 In some embodiments, sampling rendering further includes pixel shading. Step 011: performing sampling rendering on the current scene to generate a display image includes:
[0075] Step 0111: Determine the sampling position of the current scene according to the frame number corresponding to the current scene;
[0076] Step 0112: Rasterize the current scene, and color the pixels of the current scene according to the sampling position of the current scene to generate a display image.
[0077] Specifically, the frame number is used to identify the order of each frame. The sampling position of the current scene is determined according to the frame number corresponding to the current scene. Different sampling positions are set for the current scene through the frame numbers of multiple consecutive frames. The current scene is then rasterized and pixel coloring is performed according to different sampling positions to generate display images under different scenes.
[0078] See also Figure 6 In some embodiments, step 0111: determining the sampling position of the current scene according to the frame number corresponding to the current scene, includes:
[0079] Step 01111: when the frame number is an even number, determine the sampling position of the current scene as the first sampling position;
[0080] Step 01112: When the frame number is an odd number, determine that the sampling position of the current scene is the second sampling position, and the first sampling position is different from the second sampling position.
[0081] Specifically, the first sampling position is different from the second sampling position. The sampling position of the current scene is determined by whether the frame number corresponding to the current scene is an odd number or an even number. When the frame number is an even number, the sampling position of the current scene is determined to be the first sampling position; when the frame number is an odd number, the sampling position of the current scene is determined to be the second sampling position.
[0082] For example, see Figure 7 , distinguish the current scene as an even frame and an odd frame by the frame number corresponding to the current scene, so as to determine whether the sampling position of the current scene is the sampling position of the first display image or the sampling position of the second display image, perform even-numbered frame rasterization on the even-numbered frame, and perform even-numbered frame coloring according to the sampling position; perform odd-numbered frame rasterization on the odd-numbered frame, and perform odd-numbered frame coloring according to the sampling position different from the even-numbered frame.
[0083] Optionally, the continuous multi-frame display image includes a first display image and a second display image, the first display image corresponds to a first sampling position, the second display image corresponds to a second sampling position, and in the diagonal direction of the pixel, there is a preset pixel displacement between the first sampling position and the second sampling position.
[0084] The preset pixel displacement may be 0.25 pixel, 0.3 pixel, 0.5 pixel, etc.
[0085] Specifically, the continuous multiple frames of displayed images include a first display image and a second display image, and the display image of each frame is the first display image or the second display image. The sampling positions of the first display image and the second display image are different, the first display image corresponds to the first sampling position, and the second display image corresponds to the second sampling position, and the sampling position corresponding to the first display image and the sampling position corresponding to the second display image have a displacement of a preset number of pixels in the diagonal direction of the pixel. When the total display time of the continuous multiple frames of the first display image and the second display image is less than the preset human eye integration time, the effect of improving the resolution can be obtained through the visual persistence characteristics of the human eye.
[0086] For example, see Figure 8 , taking the sampling position corresponding to the first display image as the original sampling position and the preset pixel as 0.5 pixel as an example for explanation, then the sampling position corresponding to the second display image is moved by a length of D toward the lower right corner of the pixel diagonal relative to the sampling position corresponding to the first display image, where D is half the diagonal length of the pixel. When the first display image and the second display image are superimposed in the human eye after sampling and rendering, a superimposed effect as shown in the figure will be displayed.
[0087] That is to say, still taking the above example, when sampling and rendering the first display image and the second display image at different positions, assuming that counting starts from the 0th frame, the sampling position of the 0th frame is set as the original sampling position, and the first display image is obtained after sampling and rendering the original sampling position, then the first frame will determine the sampling position at a displacement position of half a pixel from the original sampling position along the pixel diagonal direction relative to the original sampling position as the sampling position of the second display image, and the second frame will determine the sampling position corresponding to the first display image of the second frame, which is still at the first sampling position. Then, when the frame number is an even number, the sampling position of the current scene is determined to be the first sampling position; when the frame number is an odd number, the sampling position of the current scene is determined to be the second sampling position. After multiple consecutive frames, the first display image or the second display image with different sampling positions is obtained, and there is a half-pixel displacement in the diagonal direction of the pixels of each consecutive sampling position of the first display image and the second display image.
[0088] Please continue reading Figure 7 , taking the preset pixel as 0.5 pixel as an example, it can be seen that, assuming that counting starts from the even frame, the sampling positions of the consecutive odd frames and even frames have a displacement of 0.5 pixels in the diagonal direction of the pixel. Due to the displacement difference between the consecutive odd frames and even frames, it is necessary to use different rendering methods for the odd frames and the even frames. After rasterization and shading processing is performed on the even frames, since the rasterization sampling position is changed in the odd frames (the position is moved by 0.5 pixels in the diagonal direction), the shading processing will be performed according to the changed sampling position in the shading processing stage, and finally the resolution of the corresponding size is determined as the output frame through Framebuffer resampling. For example, assuming that the resolution of the LCD display is n*n, according to the acquired frame signal, the even frames are rasterized and shaded to obtain the first display image, and the sampling positions after rasterization of the odd frames are shaded to obtain the second display image, and the output of the display image with n*n resolution is determined to be output to the display through Framebuffer resampling. Compared with the current sampling rendering method, more details can be sampled.
[0089] The continuous multiple frames are divided into even frames and odd frames to perform sampling rendering of the first display image and the second display image. After the sampling rendering is completed, when the generated display image is displayed by the display, the polarization angle of the display light corresponding to each frame of the display image displayed by the display will be adjusted based on the optical module. The optical module may include a polarization modulator, a birefringent crystal and a Fresnel lens arranged in sequence. The birefringent crystal may be a quartz crystal, calcite, etc. The embodiment of the present application takes the birefringent crystal as a quartz crystal as an example to illustrate the principle of the optical path. The S linear polarized light emitted by the LCD enters the polarization modulator, and the light can be modulated into 0° or 90° linear polarized light after passing through the polarization modulator. The 0° linear polarized light propagates in the manner of o light after passing through the quartz crystal, and the 90° linear polarized light propagates in the manner of e light after passing through the quartz crystal. Therefore, the projected images of o light and e light will have a half-pixel offset after passing through the quartz crystal. Based on the above optical path principle, even frames and odd frames can be distinguished according to the frame number and conveyed to the FPGA. The FPGA controls the signal generator to send the corresponding control voltage to change the modulation state of the polarization modulator. In the even frame, the polarization modulator is used to modulate the light into 0° linear polarized light so that it is emitted along the original path. In the odd frame, the polarization modulator is used to modulate the light into 90° linear polarized light so that it is emitted with a half-pixel offset. After the emitted light passes through the quartz crystal, it passes through the Fresnel lens and forms an image in the human eye.
[0090] Then, the thickness d of the quartz crystal should be designed as:
[0091]
[0092] Where L is the width of a pixel on the display, Φ is the dispersion angle of e-light, and the solution is:
[0093]
[0094] Where θ is the angle between the incident light and the optical axis, n o is the refractive index of o light, n e is the refractive index of e-light.
[0095] For example, see Fig. 9, when two frames are superimposed, even frames and odd frames can be distinguished according to the frame number. Assume that counting starts from the 0th frame (even frame), the 0th frame (even frame) and the 1st frame (odd frame) are sampled and rendered, and two consecutive frames are superimposed and imaged in the human eye. Set the sampling position of the 0th frame to the original sampling position, obtain the first display image after sampling and rendering the original sampling position, control the FPGA to send the corresponding gate circuit signal to control the signal generator, and the signal generator sends a control voltage according to the received signal to control the change of the state of the polarization modulator, so that the first display image of the 0th frame enters the polarization modulator through the display light (S linear polarized light) emitted by the display, and becomes 0° linear polarized light. After the 0° linear polarized light passes through the quartz crystal, it propagates from the original light emission path in the way of o light through the Fresnel lens and enters the human eye; determine the sampling position of the first display image along the 0th frame The sampling position of the displacement position of 0.5 pixels in the diagonal direction of the image is the sampling position of the first frame. After the sampling rendering is completed, the second display image is generated, and the FPGA is controlled to send a corresponding gate circuit signal to control the signal generator. The signal generator sends a control voltage according to the received signal to control the change of the state of the polarization modulator, so that the second display image of the first frame, the display light (S linear polarized light) emitted by the display enters the polarization modulator and becomes 90° linear polarized light. After passing through the quartz crystal, the 90° linear polarized light is emitted from the position offset by half a pixel, propagates through the Fresnel lens in the form of e-light and enters the human eye.
[0096] When performing three-frame superposition, it is assumed that counting starts from the 0th frame (even frame), that is, the 0th frame (even frame), the 1st frame (odd frame) and the 2nd frame (even frame) are sampled and rendered, and two consecutive frames are superimposed and imaged in the human eye. The adjustment of the display light of the display images of the 0th frame and the 1st frame after passing through the optical module is as described above, and no further details are given here. When sampling the second frame, the sampling position of the first display image along the 0th frame is determined as the sampling position of the second frame. After the sampling and rendering are completed, the first display image is generated. The display light of the first display image of the second frame will be adjusted to 0° linear polarized light by the polarization modulator. After passing through the quartz crystal, it will propagate from the original light emission path that is 0.5 pixels offset from the first frame, and pass through the Fresnel lens into the human eye in the manner of o light. Finally, the three frames of display images are superimposed in the human eye.
[0097] From the above, we can see that the o-light is emitted from the original path, and the e-light is emitted with a half-pixel offset relative to the o-light. The dispersion angle Φ of the e-light is:
[0098]
[0099] Φ and the angle θ between the incident light and the optical axis, o the refractive index of the light n o and the refractive index n of e-light e Related.
[0100] It is understandable that the rotation angle of the birefringent crystal around the optical axis can be adjusted by adding a mechanical device to shift the outgoing light in different directions, and combined with the corresponding sampling rendering position, higher resolution enhancement can be achieved.
[0101] It should be pointed out that quartz crystal has a birefringent property. When light passes through quartz crystal, refraction and optical path difference will change, which will affect the light propagation path and imaging effect. The change of optical path difference will cause a slight shift in the light propagation path when raster sampling is performed at different positions, which will affect the imaging of the sampling position after the pixel is shifted. Therefore, in order to obtain an accurate imaging effect, the displacement of the preset number of pixels between the first sampling position and the second sampling position can be determined according to the thickness of the quartz crystal in actual applications.
[0102] In addition, the optical module can also be a polarization modulator, a birefringent crystal and a folded optical path component, wherein the folded optical path component can be a pancake component, and the polarization modulator, birefringent crystal and pancake component are arranged in sequence along the emitting direction of the display light. The principles are similar and will not be elaborated here.
[0103] See also Fig.10 Compared with the sampling rendering method described above, in which sampling and shading are performed without distinguishing between odd and even frames during sampling and rendering, but sampling and rendering are performed twice at different positions on the same scene in one frame (i.e., a single-frame self-superposition method) to improve the resolution of the entire scene in each frame, the sampling rendering provided in the embodiment of the present application performs rasterization and shading processing on different scenes at different positions in odd and even frames (i.e., an inter-frame superposition method). At the same frame rate, the number of scenes displayed by this solution is greater than the number of scenes displayed by the single-frame self-superposition method, which can achieve sampling of more details and improve the resolution without reducing the frame rate.
[0104] This is because, in static scenes, the inter-frame superposition method of the present application has the same resolution improvement effect as the single-frame superposition method, and both methods will not lose frame rate; in dynamic scenes, when the objects in the current scene change, and the changes occur within the integration time of the human eye, the human eye cannot distinguish the actual objects displayed in each frame, and it is difficult to track the changes of the object in real time. The role of resolution is no longer reflected. At this time, the effect of improving the resolution of each frame by single-frame self-superposition cannot be reflected.
[0105] Since the human eye has an area of focus, namely the area where the human eye tracks, when the human eye tracks a moving object in real time, the relative position between the human eye and the moving object does not change, namely the human eye and the moving object are relatively still. Then the visual effect of the moving object tracked in real time in the human eye is very close to the superposition effect when the motion is compensated back to the previous frame position. Therefore, for the moving object in the area tracked by the human eye, there will be a resolution improvement effect similar to the single-frame self-superposition method; while in the area not tracked by the human eye, the resolution will be greater than or equal to the original resolution, which is in line with the usage requirements of the display of the VR device.
[0106] Therefore, it can be understood that when users use VR devices, the resolution requirements for the area tracked by the human eye are higher, and the resolution requirements for the area not tracked by the human eye are not high.
[0107] For example, see Fig.11 , when the human eye maintains the real-time tracking state in the figure and the GPU renders the moving scene and the static scene respectively, the human eye always looks at the shadow block in the scene. At this time, the shadow block in the moving scene is tracked by the human eye in real time, and the resolution is also enhanced. When calculating the resolution, the relative position d1 of the human eye and the moving scene has not changed, that is, the human eye and the shadow object in the moving scene are relatively still, so that the motion compensation d can be automatically realized by the human eye. Therefore, the resolution can be improved through the visual persistence of the human eye, and the resolution of the moving object tracked by the human eye in real time in the tracking area of the human eye can be improved; while in the static scene, there is no motion compensation for the area tracked by the human eye, and the resolution can be improved.
[0108] In summary, through the display method of the present application, the resolution of the tracked objects within the dynamic range of the human eye is improved to a certain extent, the resolution of the remaining areas is maintained, and the frame rate is kept stable. Compared with the current single-frame self-superposition method, when displaying multiple scenes continuously, the ghosting is reduced and the dynamic effect is improved.
[0109] See also Fig.12 In some embodiments, sampling rendering further includes vertex processing. In step 011, sampling rendering is performed on the current scene to generate a display image, and further includes:
[0110] Step 0113: Perform vertex processing on the current scene to determine multiple rendering areas to be sampled;
[0111] Step 0112: rasterizing the current scene and performing pixel shading on the current scene according to the sampling position of the current scene to generate a display image, further comprising:
[0112] Step 01121: determining a to-be-sampled rendering area matched by each pixel of the current pixel array, the current pixel array being determined according to a preset sampling density;
[0113] Step 01122: Based on the sampling position of the current scene, perform pixel shading on pixels that match the sampled rendering area to generate a display image.
[0114] Specifically, when converting a three-dimensional image to a two-dimensional screen, it is necessary to perform coordinate transformation operations on the vertices of the polygon. By using linear algebra calculations, the coordinate data of each vertex in the three-dimensional coordinates is converted and mapped to the two-dimensional space, thereby determining multiple areas (primitives) to be sampled and rendered in the current scene, and then determining the pixel array through a preset sampling density (such as determining the preset sampling density based on the resolution of the display), generating fragments corresponding to the screen pixel points covered by the primitives, and finally, through different sampling positions of the current scene, coloring the matching fragments in the area to be sampled and rendered, and generating display image pixels after a series of tests.
[0115] See also Fig.13 In order to better implement the display method of the embodiment of the present application, the embodiment of the present application also provides a display device 10, including a generation module 11 and a display module 11. The generation module 11 is used to perform sampling rendering on the current scene to generate a display image. The sampling rendering includes rasterization. The sampling positions of consecutive frames of display images are different during rasterization. The display module 11 is used to display the display image and adjust the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene.
[0116] In some embodiments, sampling rendering also includes pixel shading, and the generation module 11 is specifically used to determine the sampling position of the current scene according to the frame number corresponding to the current scene; rasterize the current scene, and perform pixel shading on the current scene according to the sampling position of the current scene to generate a display image.
[0117] In some embodiments, the generation module 11 is specifically used to determine the sampling position of the current scene as the first sampling position when the frame number is an even number; when the frame number is an odd number, determine the sampling position of the current scene as the second sampling position, and the first sampling position and the second sampling position are different.
[0118] In some embodiments, sampling rendering also includes vertex processing, and the generation module 11 is specifically used to perform vertex processing on the current scene to determine multiple rendering areas to be sampled; determine the rendering areas to be sampled that match each pixel of the current pixel array, and the current pixel array is determined according to a preset sampling density; according to the sampling position of the current scene, pixel coloring is performed on the pixels matching the rendering areas to be sampled to generate a display image.
[0119] In some embodiments, the display device also includes a synchronization module 13, which is used to send a current frame signal to the sampling rendering module and the display control module respectively to synchronize the sampling rendering module and the display control module. The sampling rendering module is configured to generate a display image, and the display control module is configured to send a control signal to the polarization modulator to control the polarization modulator to adjust the polarization angle of the display light. When the sampling rendering module and the display control module are synchronized, the polarization angle matches the sampling position corresponding to the current scene.
[0120] The display device 10 is described above from the perspective of functional modules in conjunction with the accompanying drawings. The functional modules can be implemented in hardware form, can be implemented in software form, or can be implemented in combination with hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by hardware integrated logic circuits in the processor and / or software form instructions, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware encoding processor, or can be executed by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiments in combination with its hardware.
[0121] See also Figure 1 The display device 100 of the embodiment of the present application includes a body 10, a processor 30, a memory 40 and a computer program, wherein the computer program is stored in the memory 40 and executed by the processor 30, and the computer program includes instructions for executing the display method of any of the above embodiments.
[0122] Optionally, the processor 30 includes a main processor 31 and a graphics processing unit (GPU) 32, and the GPU processor 32 performs sampling and rendering on the current scene to generate a scene image.
[0123] Optionally, the display device 100 also includes a display 20, a polarization modulator, a sampling and rendering module 50 and a display control module 60, the display 20 is configured to display a display image, the polarization modulator is configured to adjust the polarization angle of the display light emitted by the display, the sampling and rendering module 50 is configured to generate a display image, and the display control module 60 is configured to send a control signal to the polarization modulator to control the polarization modulator to adjust the polarization angle of the display light, and send a frame signal through the main processor 31 to synchronize the sampling and rendering module 50 and the display control module 60, so that the display image and the polarization angle of the light modulated by the polarization modulator can correspond.
[0124] Optionally, the display device 100 also includes an optical module, which includes a polarization modulator, a birefringent crystal and a Fresnel lens, and the polarization modulator, the birefringent crystal and the Fresnel lens are arranged in sequence along the exit direction of the display light; or, the optical module includes a polarization modulator, a birefringent crystal and a folded optical path component, and the polarization modulator, the birefringent crystal and the folded optical path component are arranged in sequence along the exit direction of the display light.
[0125] See also Figure 1 and Fig.14 In the case where the current scene is sampled and rendered in the GPU image processor 32 to generate a scene image and the display 20 is controlled by the FPGA, the main processor synchronously sends a frame signal to the GPU image processor 32 and the FPGA. In the sampling and rendering module 50, the GPU image processor 32 performs vertex processing and rasterization according to the application scenario, and generates a display image after performing pixel shading processing and Framebuffer resampling processing according to the frame signal. At the same time, after receiving the frame signal, the FPGA sends a gate circuit signal to control the signal generator. The signal generator sends a control voltage to change the state of the polarization modulator according to the received signal. The GPU image processor 32 sends the generated display image to the display 20 for display. The display light emitted by the display 20 is modulated into 0° or 90° linear polarized light through the polarization modulator. The 0° or 90° linear polarized light passing through the polarization modulator is emitted as o light or e light through the birefringent crystal. The o light or e light emitted through the birefringent crystal is superimposed on the human eye after passing through the optical module. The pixel shading processing and Framebuffer resampling processing of the GPU are continuously switched according to the frame signal, and the state of the polarization modulator is synchronously selected, so that the resolution is improved.
[0126] See also Fig.15 The embodiment of the present application also provides a computer-readable storage medium 600 on which a computer program 610 is stored. When the computer program 610 is executed by a processor 620, the steps of the display method of any of the above-mentioned embodiments are implemented. For the sake of brevity, they are not repeated here.
[0127] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0128] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0129] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A display method, characterized in that: include: Performing sampling rendering on the current scene to generate a display image, wherein the sampling rendering includes rasterization, and the sampling positions of the display images of consecutive frames are different when rasterization is performed; The display image is displayed, and the polarization angle of the display light corresponding to the display image is adjusted so that the polarization angle matches the sampling position corresponding to the current scene, and different polarization angles correspond to different display positions of the display image.
2. The display method according to claim 1, characterized in that: The sampling rendering further includes pixel shading, and the sampling rendering of the current scene to generate a display image includes: Determining the sampling position of the current scene according to the frame number corresponding to the current scene; The current scene is rasterized, and the pixel shading is performed on the current scene according to the sampling position of the current scene to generate the display image.
3. The display method according to claim 2, characterized in that: The step of determining the sampling position of the current scene according to the frame sequence number corresponding to the current scene includes: When the frame sequence number is an even number, determining the sampling position of the current scene as the first sampling position; When the frame sequence number is an odd number, the sampling position of the current scene is determined to be a second sampling position, and the first sampling position is different from the second sampling position.
4. The display method according to claim 3, characterized in that: The continuous multiple frames of display images include a first display image and a second display image, the first display image corresponds to the first sampling position, the second display image corresponds to the second sampling position, and in the diagonal direction of the pixel, there is a preset pixel displacement between the first sampling position and the second sampling position.
5. The display method according to claim 3, characterized in that: The sampling rendering also includes vertex processing, and the sampling rendering of the current scene to generate a display image also includes: Performing vertex processing on the current scene to determine a plurality of rendering areas to be sampled; The rasterizing the current scene and performing pixel shading on the current scene according to the sampling position of the current scene to generate the display image includes: Determine the rendering area to be sampled that matches each pixel of the current pixel array, wherein the current pixel array is determined according to a preset sampling density; According to the sampling position of the current scene, the pixel shading is performed on the pixels matching the to-be-sampled rendering area to generate the display image.
6. The display method according to claim 1, characterized in that: Also includes: Sending a current frame signal to a sampling rendering module and a display control module respectively to synchronize the sampling rendering module and the display control module, wherein the sampling rendering module is configured to generate the display image, and the display control module is configured to control the polarization modulator to adjust the polarization angle of the display light by sending a control signal to the polarization modulator; When the sampling rendering module and the display control module are synchronized, the polarization angle matches the sampling position corresponding to the current scene.
7. The display method according to claim 1, characterized in that: The total display time of the continuous multiple frames of the displayed image is less than the preset human eye integration time.
8. A display device, characterized in that: The display device comprises: A generation module, used for performing sampling rendering on the current scene to generate a display image, wherein the sampling rendering includes rasterization, and the sampling positions of consecutive frames of the display image are different when rasterization is performed; The display module is used to display the display image and adjust the polarization angle of the display light corresponding to the display image so that the polarization angle matches the sampling position corresponding to the current scene.
9. A display device, characterized in that: include: Processor, memory; and A computer program, wherein the computer program is stored in the memory and executed by the processor, and the computer program includes instructions for executing the display method according to any one of claims 1 to 7.
10. The display device according to claim 9, characterized in that Also includes: a display configured to display the display image; a polarization modulator configured to adjust the polarization angle of display light emitted by the display; A sampling and rendering module, configured to generate the display image; The display control module is configured to control the polarization modulator to adjust the polarization angle of the display light by sending a control signal to the polarization modulator.
11. The display device according to claim 9, characterized in that It also includes an optical module, which includes the polarization modulator, the birefringent crystal and the Fresnel lens, and the polarization modulator, the birefringent crystal and the Fresnel lens are arranged in sequence along the exit direction of the display light; or, the optical module includes the polarization modulator, the birefringent crystal and the folded optical path component, and the polarization modulator, the birefringent crystal and the folded optical path component are arranged in sequence along the exit direction of the display light.
12. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the display method according to any one of claims 1 to 7.