Display method, display device, display equipment and storage medium

Abstract

According to the display method, the display device, the display equipment and the non-volatile computer readable storage medium, sampling rendering is carried out on the current scene to generate the display image, the current scene comprises the multiple to-be-sampled rendering areas, the to-be-sampled rendering areas have the matched M to-be-rendered pixels, and the M to-be-rendered pixels correspond to the M to-be-rendered pixels. The display image is generated by sampling and rendering according to the position of a target to-be-rendered pixel in the M to-be-rendered pixels, the positions of the target to-be-rendered pixels corresponding to continuous M frames of display images are different from each other, and M is a positive integer; a target display pixel of a display screen is controlled to work so as to display a display image, the display screen comprises a plurality of pixel areas, each pixel area comprises M display pixels, and the target display pixel is a display pixel corresponding to the position of a target to-be-rendered pixel in the M display pixels, so that the display positions of continuous M frames of display images displayed by the display screen are different, the frame rate is effectively improved, and the use experience of the user when using the VR equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and more specifically, to a display method, a display device, a display equipment, and a non-volatile computer-readable storage medium. Background Art

[0002] The frame rate and resolution of virtual reality (VR) devices are important factors that affect the user's viewing experience. The frame rate affects the smoothness of the user's viewing, and the frame rate is closely related to the rendering speed of the picture. When the rendering speed is slow, the frame rate of the VR device will not be high enough, causing the picture seen by the user to freeze, thus affecting the user's experience and immersion when using VR devices. Summary of the invention

[0003] The embodiments of the present application provide a display method, a display device, a display equipment and a non-volatile computer-readable storage medium, which will sample and render the current scene, generate a display image, and control the target display pixels of the display screen to operate so as to display the display image accordingly. It can effectively reduce the sampling and rendering time and increase the frame rate while maintaining the stability of the resolution, thereby improving the user's viewing experience.

[0004] The display method of the embodiment of the present application includes: sampling and rendering a current scene to generate a display image, the current scene includes a plurality of to-be-sampled rendering areas, the to-be-sampled rendering areas have M matching to-be-rendered pixels, the display image is generated by sampling and rendering according to the position of a target to-be-rendered pixel among the M to-be-rendered pixels, the positions of the target to-be-rendered pixels corresponding to the display images of M consecutive frames are different, and M is a positive integer; controlling a target display pixel of a display screen to work to display the display image, the display screen includes a plurality of pixel areas, the pixel area includes M display pixels, and the target display pixel is a display pixel among the M display pixels corresponding to the position of the target to-be-rendered pixel.

[0005] In some embodiments, each of the to-be-sampled rendering areas has matching M pixels to be rendered in the pixel array corresponding to the original sampling density, and the original sampling density corresponds to the resolution of the display screen; the sampling rendering includes rasterization and pixel shading, and the sampling rendering of the current scene to generate a display image includes: determining the target pixel to be rendered corresponding to the current scene according to the frame number of the current scene; and sequentially rasterizing and pixel shading the positions of the target pixel to be rendered corresponding to the current scene to generate the display image, and the rasterization sampling density is 1 / M of the original sampling density.

[0006] In some embodiments, determining the target pixel to be rendered corresponding to the current scene according to the frame number of the current scene includes: dividing the frame number of the current scene by M to determine a remainder; and determining the target pixel to be rendered corresponding to the current scene according to the remainder.

[0007] In some embodiments, the sampling rendering further includes vertex processing, and the sampling rendering of the current scene to generate a display image further includes: performing vertex processing on the current scene to determine a plurality of the areas to be sampled and rendered.

[0008] In some embodiments, it also includes: sending a current frame signal to a sampling rendering module and a display control module respectively to synchronize the sampling rendering module and the display control module, the sampling rendering module is configured to generate the display image, and the display control module is configured to control the operation of the target display pixel of the display screen by sending a control signal to the display screen; after the sampling rendering module and the display control module are synchronized, the target pixel to be rendered corresponding to the display image corresponds to the target display pixel position on which the display screen is operating.

[0009] In some embodiments, the total display time of the M consecutive frames of the displayed image is less than a preset human eye integration time.

[0010] In some embodiments, M is equal to 4, the target pixel to be rendered includes 1 pixel to be rendered, and the 4 pixels to be rendered are arranged in a matrix.

[0011] The display device of the present application includes a generation module and a display module. The generation module is used to sample and render the current scene to generate a display image, the current scene includes multiple sampling and rendering areas, the sampling and rendering areas have M matching pixels to be rendered, the display image is sampled and rendered according to the position of the target pixel to be rendered among the M pixels to be rendered, the positions of the target pixel to be rendered corresponding to the display images of M consecutive frames are different, and M is a positive integer; the display module is used to control the target display pixel of the display screen to work to display the display image, the display screen includes multiple pixel areas, the pixel area includes M display pixels, and the target display pixel is the display pixel among the M display pixels corresponding to the position of the target pixel to be rendered.

[0012] The display device of the present application includes a processor, a memory, a display, and a computer program, wherein the computer program is stored in the memory and executed by the processor, and the computer program includes instructions for executing the display method described in any of the above embodiments.

[0013] In some embodiments, the display device of the present application further includes a sampling rendering module and a display control module, wherein the sampling rendering module is configured to generate the display image, and the display control module is configured to control the operation of the target display pixel of the display screen by sending a control signal to the display screen.

[0014] In certain embodiments, the display device of the present application further includes an optical module, wherein the optical module includes a phase retarder and a Fresnel lens, or the optical module includes a phase retarder and a folded optical path component.

[0015] The non-volatile computer-readable storage medium of the embodiment of the present application includes a computer program. When the computer program is executed by a processor, the processor executes the display method described in any of the above embodiments.

[0016] The display method, display device, display equipment and non-volatile computer-readable storage medium of the embodiment of the present application will sample and render the current scene to generate a display image, and control the target display pixel of the display screen to work to display the display image. In the sampling and rendering processing process of M consecutive frames, the display image is sampled and rendered according to the position of the target pixel to be rendered among the M pixels to be rendered, and only the target pixel to be rendered in the sampling rendering area is sampled and rendered, and the non-target pixel to be rendered is not processed, which can reduce the sampling and rendering time of each frame of the sampling rendering area, thereby improving the sampling and rendering speed of the current scene, improving the efficiency of graphics processing, and achieving an increase in frame rate; and, by making a one-to-one correspondence between the position of the target display pixel of the display screen and the position of the target pixel to be rendered in the sampling rendering area, the target display pixel of the display screen is controlled to be the target pixel to be rendered corresponding to each frame, so that the display positions of the consecutive M frames of display images displayed by the display screen are different, and the resolution of the consecutive M frames of display images is realized. Superimposed imaging in the human eye, thereby not only improving the frame rate, but also ensuring that the resolution is not lost, and improving the user experience when using VR devices.

[0017] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0019] Figure 1 is a schematic flow chart of a display method of certain embodiments of the present application;

[0020] Figure 2It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0021] Figure 3 is a schematic flow chart of a display method of certain embodiments of the present application;

[0022] Figure 4 is a schematic flow chart of a display method of certain embodiments of the present application;

[0023] Figure 5 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0024] Figure 6 is a schematic flow chart of a display method of certain embodiments of the present application;

[0025] Figure 7 is a schematic flow chart of a display method of certain embodiments of the present application;

[0026] Figure 8 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0027] Fig. 9 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0028] Fig.10 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0029] Fig.11 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0030] Fig.12 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0031] Fig.13 is a schematic diagram of a module of a display device according to some embodiments of the present application;

[0032] Fig.14 is a schematic diagram of the structure of a display device according to some embodiments of the present application;

[0033] Fig.15 It is a scene schematic diagram of a display method of certain embodiments of the present application;

[0034] Fig.16 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0036] To facilitate understanding of this application, the terms appearing in this application are explained below:

[0037] 1. Virtual reality technology (VR), also known as virtual reality or spiritual technology. Virtual reality technology encompasses computers, electronic information, and simulation technology. Its basic implementation method is based on computer technology, utilizing and integrating the latest developments in a variety of high technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. The graphics processor (GPU) in the VR device processes the image in the current scene to produce a realistic three-dimensional visual, tactile, olfactory and other sensory experiences in the virtual world, so that people in the virtual world have an immersive feeling. The technical solution provided in the embodiment of the present application is mainly used to enhance the frame rate of the GPU processing image of the VR device, and the display method is applied to the display, effectively improving the frame rate while maintaining the stability of the display resolution.

[0038] 2. Frame rate refers to the number of frames played per second when a video is played, expressed in frames per second (fps). In VR technology, it is generally believed that the higher the frame rate, the smoother the picture and the better the user experience.

[0039] 3. Vertex processing: The vertices of 3D graphics have a three-dimensional coordinate. Through linear algebra calculations, the coordinate data of each vertex in the three-dimensional space is converted and drawn into the two-dimensional space of the display. At the same time, the color of the vertex is calculated for subsequent pixel color interpolation. This operation is called vertex processing.

[0040] 4. Rasterization: The primitive information generated during vertex processing is input into the rasterization stage. Primitives are assembled first, and then the fragments corresponding to the screen pixels covered by the primitives are determined through triangle traversal. After the rasterization stage, the primitives will be divided into basic units of pixel size, which are called fragments. Fragments are more like data representations of pixels. The final pixels are generated by the information in the fragments, and then the fragments are colored by the fragment shader. When operating fragment by fragment, the fragments undergo a series of test processes. The tested fragments will be converted into pixels and finally presented in the frame buffer area.

[0041] 5. Field Programmable Gate Array (FPGA) is a programmable logic device, which is a semiconductor chip composed of a series of programmable logic gates. In the technical solution provided in the embodiment of the present application, it is mainly used to send gate circuit signals to control the display screen to control the target display pixels that need to work to display.

[0042] 6. Framebuffer: A video output device drives a video display device from a memory buffer containing complete frame data. The technical solution provided in the embodiment of the present application is mainly used to generate a display image after determining the required resolution as the output frame through Framebuffer resampling.

[0043] At present, the frame rate of VR devices is mainly affected by the performance of the display and the speed of sampling and rendering the current scene. The performance of the display is mainly limited by the processing technology. Traditional displays use the interlaced display method to display odd or even rows of pixels each time to reduce the bandwidth of each transmission to increase the display frame rate. The current sampling rendering is generally rasterized according to the sampling density of 2n*2n, and after all the rasterized pixels are colored, they are directly output to the display for display. The existing pixel shifting and superposition method based on time division multiplexing applied to VR devices achieves resolution improvement by continuously sampling and rendering the same scene multiple times, and superimposing multiple display images of the same scene through the visual persistence characteristics of the human eye.

[0044] Although the resolution increase method based on time-division multiplexing can break through the upper limit of the display's resolution, it will reduce the frame rate and the dynamic performance of the VR device when it is working, thus affecting the user's viewing experience.

[0045] In order to solve the above technical problems, the embodiment of the present application provides a display method, which will be described in detail below:

[0046] See also Figure 1, an embodiment of the present application provides a display method, the display method comprising:

[0047] Step 011: Sampling and rendering the current scene to generate a display image, the current scene includes a plurality of to-be-sampled rendering areas, the to-be-sampled rendering areas have M matching to-be-rendered pixels, the display image is generated by sampling and rendering according to the position of a target to-be-rendered pixel among the M to-be-rendered pixels, the positions of the target to-be-rendered pixels corresponding to the display images of M consecutive frames are different, and M is a positive integer;

[0048] The current scene is a frame of multiple frames of scenes included in the scene data stream, and the scene data stream is generated by a virtual three-dimensional (3D) scene or a two-dimensional scene built by an application.

[0049] Specifically, taking the current scene as a 3D scene as an example, the GPU of the VR device performs view transformation on the current scene, converts the three-dimensional coordinates of the 3D scene graph into two-dimensional coordinates, determines multiple rendering areas (primitives) to be sampled in the current scene, and there are M matching pixels to be rendered in the rendering areas to be sampled. Then, sampling and shading and a series of test sampling rendering are performed on the target pixel to be rendered among the M pixels to be rendered in the rendering areas to be sampled (that is, the position of the target pixel to be rendered is used as the sampling position), and finally a display image is generated.

[0050] For example, vertex processing can be used to perform view transformation on the current scene in the scene data stream, determine multiple rendering areas to be sampled in the current scene, determine a preset sampling density based on the resolution of the display, determine a pixel array based on the preset sampling density, match the pixel array with the rendering area to be sampled through rasterization, and finally, based on information of the rendering area to be sampled (such as texture, color, etc.) and the preset sampling position, shade the pixels in the rendering area to be sampled to obtain corresponding pixel values, thereby generating a display image.

[0051] It is determined that the sampling and rendering of the current area to be sampled and rendered needs to be completed through M consecutive frames. The positions of the target pixels to be rendered corresponding to each display image of the consecutive M frames are different from each other. Then, after completing the continuous sampling of M frames, all M pixels to be rendered will be sampled and rendered, thereby realizing the sampling and rendering of all pixels to be rendered in the current area to be sampled and rendered.

[0052] It is understandable that M is a positive integer. When one pixel to be rendered is sampled in each frame, that is, the target pixel to be rendered is one pixel to be rendered, there are M target pixels to be rendered, and M sampling is required to achieve sampling rendering of the current rendering area to be sampled.

[0053] For example, see the scenario Figure 2, obtain the current scene in the scene data stream, in the to-be-sampled rendering area of ​​the current scene, there is a 2*2 pixel array, which includes 4 1*1 to-be-rendered pixels, that is, M=4, and the to-be-rendered area needs to be sampled for four consecutive frames, and each frame samples and renders a 1*1 target to-be-rendered pixel. The four consecutive frames include the first frame, the second frame, the third frame and the fourth frame. The target to-be-rendered pixel of the first frame is the to-be-rendered pixel in the upper left corner of the 2*2 matrix formed by the 4 to-be-rendered pixels; the target to-be-rendered pixel of the second frame is the to-be-rendered pixel in the upper right corner of the 2*2 matrix formed by the 4 to-be-rendered pixels; the target to-be-rendered pixel of the third frame is the to-be-rendered pixel in the lower left corner of the 2*2 matrix formed by the 4 to-be-rendered pixels; the target to-be-rendered pixel of the fourth frame is the to-be-rendered pixel in the lower right corner of the 2*2 matrix formed by the 4 to-be-rendered pixels. The positions of the target pixels to be rendered of four consecutive frames of sampling rendering are different. After completing the sampling rendering of the first frame, the second frame, the third frame and the fourth frame, the four pixels to be rendered in the sampling rendering area will all be sampled and rendered, and a display image will be generated.

[0054] Optionally, the total display time of the M consecutive frames of displayed images is less than a preset human eye integration time.

[0055] Specifically, it takes a certain integration time for an image to be formed on the retina, and the inter-frame images will be superimposed on the retina during the integration time. Due to the visual persistence characteristics of the human eye, the human eye obtains the superimposed images during the integration time. When the total display time of M consecutive frames is less than the preset human eye integration time, the resolution perceived by the human eye will exceed the original resolution of the screen, thereby achieving the effect of improving the resolution.

[0056] For example, in general, the daytime integration time of the human eye is 50 milliseconds (ms). The frame rate of the VR device display is 90 Hz, so the interval between frames is 1 second (s) / 90HZ=0.01111s=11.11ms, and the time to display four consecutive frames is 11.11ms*4=44.44ms, which is less than the daytime integration time of the human eye of 50ms. Therefore, inter-frame imaging can be superimposed on the retina, so that the resolution perceived by the human eye is close to the original resolution of the screen, thereby achieving the effect of improving resolution.

[0057] Step 012: Control the target display pixel of the display screen to work to display the display image, the display screen includes multiple pixel areas, the pixel area includes M display pixels, and the target display pixel is a display pixel among the M display pixels corresponding to the position of the target pixel to be rendered.

[0058] Specifically, the display includes multiple pixel areas, and there are M display pixels corresponding to the pixels to be rendered in the pixel areas. By making a one-to-one correspondence between the positions of the target display pixels of the display and the positions of the target pixels to be rendered in the sampling and rendering areas, after the target pixels to be rendered that have completed sampling and rendering generate a display image, the target display pixels on the display can be controlled to be displayed accordingly, so as to control other pixels that have not been sampled and rendered to not be displayed, so that the display image generated after sampling and rendering the target pixels to be rendered in each frame can correspond to the pixels controlled by the display each time.

[0059] In this way, during the sampling and rendering processing of M consecutive frames of the VR device, the display image is sampled and rendered according to the position of the target pixel to be rendered among the M pixels to be rendered, and only the target pixel to be rendered in the sampled rendering area is sampled and rendered, and the non-target pixel to be rendered is not processed, which can reduce the sampling and rendering time of the sampled rendering area for each frame, thereby improving the sampling and rendering speed of the current scene, improving the efficiency of graphics processing, and achieving an increase in frame rate; and, by making a one-to-one correspondence between the position of the target display pixel of the display screen and the position of the target pixel to be rendered in the sampled rendering area, the target display pixel of the control display screen is controlled to be the target pixel to be rendered corresponding to each frame each time, so that the display positions of the consecutive M frames of display images displayed on the display screen are different, so that the resolution of the consecutive M frames of display images is superimposed in the human eye, thereby not only improving the frame rate, but also ensuring that the resolution is not lost, thereby improving the user experience when using the VR device.

[0060] See also Figure 3 In some embodiments, each to-be-sampled rendering area has M matching to-be-rendered pixels in a pixel array corresponding to an original sampling density, and the original sampling density corresponds to a resolution of a display screen; sampling rendering includes rasterization and pixel shading, step 011: performing sampling rendering on the current scene to generate a display image, including:

[0061] Step 0111: Determine the target pixel to be rendered corresponding to the current scene according to the frame number of the current scene;

[0062] Step 0112: Rasterize and color the target pixel positions to be rendered corresponding to the current scene in sequence to generate a display image. The rasterization sampling density is 1 / M of the original sampling density.

[0063] The rasterized sampling density is 1 / M of the original sampling density.

[0064] Specifically, sampling rendering includes rasterization and pixel shading. Each to-be-sampled rendering area has M matching to-be-rendered pixels in the pixel array corresponding to the original sampling density, and the original sampling density corresponds to the resolution of the display screen. The target to-be-rendered pixels corresponding to the current scene are determined by the frame number of the current scene, such as by setting a mapping relationship between the parity of the frame number of the current scene and the target to-be-rendered pixels, or according to a mapping relationship between the remainder of dividing the frame number of the current scene by M and the target to-be-rendered pixels.

[0065] Since the original pixel information needs to be sampled, the original sampling density does not change. Each to-be-sampled rendering area has M matching to-be-rendered pixels in the pixel array corresponding to the original sampling density, and the original sampling density corresponds to the resolution of the display. Therefore, when the current to-be-sampled rendering area is sampled and rendered for M consecutive frames, the positions corresponding to the target to-be-rendered pixels among the M to-be-rendered pixels are rasterized and pixel shaded in sequence. At this time, the rasterized sampling density is 1 / M of the original sampling density.

[0066] For example, when the resolution of the LCD is 2n*2n, the current rendering area to be sampled includes 4 pixels to be rendered, and the original sampling density is 2n*2n pixels of the current rendering area to be sampled, corresponding to the resolution of the display. When 4 frames are sampled and rendered continuously, the rasterization of these 4 frames is only for the target pixels to be sampled and rendered, that is, only 1 n*n pixel is rasterized in each frame, then the rasterization sampling density is 1 / 4 of the original sampling density.

[0067] See also Figure 4 In some embodiments, step 0111: determining the target pixel to be rendered corresponding to the current scene according to the frame number of the current scene, includes:

[0068] Step 01111: Divide the frame number of the current scene by M to determine the remainder; and

[0069] The target pixel to be rendered corresponding to the current scene is determined according to the remainder.

[0070] Specifically, the frame number of the current scene is divided by the number of pixels to be rendered to obtain a remainder, and the remainder is mapped to the sampling position of the target pixel to be rendered in the current scene, so as to determine the target pixel to be rendered corresponding to the current scene according to the remainder. For example, if there are M matching pixels to be rendered in the to-be-sampled rendering area of ​​the current scene, each pixel to be rendered is numbered from 0 to M in sequence, and the frame number is divided by the number, and the remainder obtained is the number of the target pixel to be rendered.

[0071] For example, see Figure 5In the area to be sampled and rendered in the current scene, there is a 2*2 pixel array, which includes 4 1*1 pixels to be rendered, that is, M=4. The four pixels to be rendered are numbered in sequence, and it is determined that the pixel to be rendered No. 0 is the pixel to be rendered in the upper left corner of the 2*2 pixel array, the pixel to be rendered No. 1 is the pixel to be rendered in the upper right corner of the 2*2 pixel array, the pixel to be rendered No. 2 is the pixel to be rendered in the lower left corner of the 2*2 pixel array, and the pixel to be rendered No. 3 is the pixel to be rendered in the lower right corner of the 2*2 pixel array. The area to be rendered is sampled for four consecutive frames, and a 1*1 target pixel to be rendered is sampled and rendered in each frame. The counting starts with the 0th frame as the starting frame. The four consecutive frames include the 0th frame, the 1st frame, the 2nd frame, and the 3rd frame. That is, the frame number of the 0th frame is 0, the frame number of the 1st frame is 1, the frame number of the 2nd frame is 2, and the frame number of the 3rd frame is 3. The frame number 0 / 4 of the 0th frame obtains a remainder of 0, then the target pixel to be rendered sampled in the 0th frame is determined to be the pixel to be rendered in the upper left corner; the frame number 1 / 4 of the 1st frame obtains a remainder of 1, then the target pixel to be rendered sampled in the 1st frame is determined to be the pixel to be rendered in the upper right corner; the frame number 2 / 4 of the 2nd frame obtains a remainder of 2, then the target pixel to be rendered sampled in the 2nd frame is determined to be the pixel to be rendered in the lower left corner; the frame number 3 / 4 of the 3rd frame obtains a remainder of 3, then the target pixel to be rendered sampled in the 3rd frame is determined to be the pixel to be rendered in the lower right corner.

[0072] See also Figure 6 In some embodiments, sampling rendering further includes vertex processing, step 011: performing sampling rendering on the current scene to generate a display image, further comprising:

[0073] Step 0113: Perform vertex processing on the current scene to determine multiple rendering areas to be sampled.

[0074] Specifically, when converting a three-dimensional image to a two-dimensional screen, it is necessary to perform coordinate transformation operations on the vertices of the polygon. By using linear algebra calculations, the coordinate data of each vertex in the three-dimensional coordinates is converted and mapped to the two-dimensional space, thereby determining multiple areas (primitives) to be sampled and rendered in the current scene, and then determining the pixel array through a preset sampling density (such as determining the preset sampling density based on the resolution of the display), generating fragments corresponding to the screen pixel points covered by the primitives, and finally, through different sampling positions of the current scene, coloring the matching fragments in the area to be sampled and rendered, and generating display image pixels after a series of tests.

[0075] See also Figure 7 In some embodiments, the display method further includes:

[0076] Step 013: sending the current frame signal to the sampling rendering module and the display control module respectively to synchronize the sampling rendering module and the display control module, wherein the sampling rendering module is configured to generate a display image, and the display control module is configured to control the target display pixel of the display screen to work by sending a control signal to the display screen;

[0077] Step 014: After the sampling rendering module and the display control module are synchronized, the target pixels to be rendered corresponding to the displayed image correspond to the target display pixel positions on which the display screen is working.

[0078] Specifically, the sampling rendering module and the display control module are synchronized by sending the current frame signal to the sampling rendering module and the display control module respectively. After receiving the current frame signal, the sampling rendering module performs sampling rendering on the current scene, such as generating a display image through sampling rendering processes such as vertex processing, rasterization, pixel shading and Framebuffer resampling. After receiving the current frame signal, the display control module sends a corresponding gate circuit signal through the FPGA to control the target display pixel in the display to perform display work. After synchronizing the sampling rendering module and the display control module, the target pixel to be rendered in the sampling rendering area of ​​the current scene in the sampling rendering module, and the position of the target display pixel to be controlled in the pixel area of ​​the display in the display control module can be one-to-one corresponding, so that the display position of the display image of the display is consistent with the sampling position during sampling rendering, so that the display position of the display image of the continuous M frames is constantly changing, so as to achieve the resolution superposition of the display image of the continuous M frames, and the resolution can be guaranteed.

[0079] For example, see the scenario Figure 2 and scenes Figure 8, the sampling and rendering module is synchronized with the display control module. After receiving the current frame signal, the sampling and rendering module performs vertex processing on the current scene and determines multiple areas to be sampled and rendered. It is determined that one of the areas to be sampled and rendered has 4 pixels to be rendered arranged in a 2*2 array, that is, M=4. The sampling and rendering module sequentially samples and renders the target pixels to be rendered corresponding to the first frame, the second frame, the third frame, and the fourth frame. After receiving the current frame signal, the display control module sends corresponding gate circuit signals through FPGA to sequentially control the target display pixels corresponding to the first frame, the second frame, the third frame, and the fourth frame in the display to perform display work. When the sampling and rendering module performs sampling and rendering on the first frame, the target pixel to be rendered on the first frame is the pixel to be rendered in the upper left corner of the 2*2 matrix formed by the 4 pixels to be rendered. The target display pixel controlled by the display control module to work on the display is the display pixel in the upper left corner of the 2*2 matrix corresponding to the area to be sampled and rendered in the display, wherein the position of the pixel to be rendered in the upper left corner of the 2*2 matrix of the area to be rendered corresponds to the display pixel in the upper left corner of the 2*2 matrix of the display;

[0080] When the sampling and rendering module performs sampling and rendering on the second frame, the target pixel to be rendered of the second frame is the pixel to be rendered in the upper right corner of the 2*2 matrix formed by the four pixels to be rendered, and the target display pixel for the display control module to control the display to work is the display pixel in the upper right corner of the 2*2 matrix corresponding to the area to be sampled and rendered in the display, wherein the position of the pixel to be rendered in the upper right corner of the 2*2 matrix of the area to be rendered corresponds to the display pixel in the upper right corner of the 2*2 matrix of the display;

[0081] When the sampling and rendering module performs sampling and rendering on the third frame, the target pixel to be rendered of the third frame is the pixel to be rendered in the lower left corner of the 2*2 matrix formed by the four pixels to be rendered, and the target display pixel for the display control module to control the display to work is the display pixel in the lower left corner of the 2*2 matrix corresponding to the area to be sampled and rendered in the display, wherein the position of the pixel to be rendered in the lower left corner of the 2*2 matrix of the area to be rendered corresponds to the display pixel in the lower left corner of the 2*2 matrix of the display;

[0082] When the sampling and rendering module performs sampling and rendering on the fourth frame, the target pixel to be rendered of the fourth frame is the pixel to be rendered in the lower right corner of the 2*2 matrix formed by the four pixels to be rendered, and the target display pixel that the display control module controls the display to work is the display pixel in the lower right corner of the 2*2 matrix corresponding to the area to be sampled and rendered in the display, wherein the position of the pixel to be rendered in the lower right corner of the 2*2 matrix of the area to be rendered corresponds to the display pixel in the lower right corner of the 2*2 matrix of the display.

[0083] In static scenes, the inter-frame superposition method can achieve the same resolution perception effect as the original image; in dynamic scenes, when the objects in the current scene change and the changes occur within the integration time of the human eye, the human eye cannot distinguish the actual objects displayed in each frame, and it is difficult to track the changes of the object in real time. At this time, the role of resolution is no longer reflected.

[0084] Since the human eye has an area of ​​focus, that is, the area where the human eye tracks, when the human eye tracks a moving object in real time, the relative position between the human eye and the moving object does not change, that is, the human eye and the moving object are relatively still. At this time, the visual effect of the moving object tracked by the human eye in real time is very close to the effect of inter-frame superposition when the motion is compensated back to the previous frame position. Then, the moving object in the area tracked by the human eye will have a resolution enhancement effect close to the original image; while in the area not tracked by the human eye, the resolution will be greater than or equal to the original resolution, which is in line with the use requirements of the display of the VR device.

[0085] Therefore, it is understandable that when a user uses a VR device, the resolution requirement for the area tracked by the human eye is higher, and the resolution requirement for the area not tracked by the human eye is not high. Fig. 9 When the human eye keeps real-time tracking, the GPU of the VR device samples and renders the static scene and the dynamic scene respectively according to the acquired current frame signal. At this time, in the dynamic scene, the human eye tracks the shadow block in the current scene. Since the shadow block is tracked by the human eye in real time, the visual persistence of the human eye can better maintain the resolution, so that when calculating the resolution, since the relative position d between the human eye and the moving object has not changed, that is, the human eye and the moving object are relatively still, at this time, motion compensation d1, d2, d3 and d4 can be automatically realized by the human eye. Then, the visual effect of the moving object tracked by the human eye in real time in the human eye is very close to the effect of inter-frame superposition when the motion compensation is returned to the previous frame position. Therefore, the effect of improving the resolution can be achieved through the visual persistence characteristics of the human eye, and the resolution of the moving object tracked by the human eye in real time in the tracking area of ​​the human eye can be improved; while in the static scene, there is no motion compensation d1, d2, d3 and d4 in the area tracked by the human eye, so the resolution can be improved.

[0086] Furthermore, it can be seen from the above that the display controls the target display pixel to work to display the display image. The display image displayed on the display will be displayed on the human eye through the optical system of the VR device. The optical system of the VR device can be a Pancake optical module, that is, an optical module consisting of two 1 / 4 phase delay plates (Quarter-Wave Plate, QWP), a semi-transparent and semi-reflective mirror (Beam Splitter, BS), and a reflective polarizer (Reflective Polarizer, RP), wherein the reflective polarizer (Reflective Polarizer, RP) is configured to reflect P polarized light and transmit S polarized light.

[0087] For example, see Fig.10 , taking the display image based on the liquid crystal display (LCD) as an example, the S linear polarized light emitted by the LCD will change the shape of the polarized light after passing through the QWP and become right-circularly polarized light (RCP). After the right-circularly polarized light RCP passes through the BS, the shape of the polarized light does not change and it remains right-circularly polarized light RCP. The right-circularly polarized light RCP passes through the QWP and becomes P linear polarized light. The P linear polarized light reaches the RP and is emitted, and the reflected light is P linear polarized light. The P linear polarized light passes through the QWP and changes back to right-circularly polarized light RCP. The right-circularly polarized light RCP is reflected by the BS and becomes left-circularly polarized light (LCP). The left-circularly polarized light LCP passes through the QWP and becomes S linear polarized light. The S linear polarized light reaches the RP and is transmitted to the human eye for display.

[0088] In addition, the optical system may also include an optical module consisting of a phase delay plate and a Fresnel lens. Based on the principle of light refraction, the curved surface of the Fresnel lens is used to fold and focus light, thereby realizing the display image on the human eye.

[0089] In summary, when the total display time of M consecutive frames of display images is less than the preset human eye integration time, the target display pixels of the display are controlled to work to display the display image, and the generated display image is displayed on the human eye through the optical module of the VR device, and the display images of M consecutive frames are imaged on the human eye through inter-frame superposition imaging, that is to say, the target rendering pixels of the sampling rendering are switched, and the target display pixels of the sampling rendering are synchronously displayed on the display, and the inter-frame imaging is performed on the human eye through the optical system, thereby maintaining the resolution while improving the system frame rate.

[0090] Optionally, M is equal to 4, the target pixel to be rendered includes 1 pixel to be rendered, and the 4 pixels to be rendered are arranged in a matrix.

[0091] Specifically, in order to achieve the effect of frame rate enhancement, the number of pixels to be rendered and the number of frames in the target area to be rendered in the current scene are determined to be 4, and the 4 pixels to be rendered are arranged in a matrix, such as a matrix of 4n*n, or a matrix of 2n*2n, etc. The sampling rendering is performed by sampling and rendering the sampled rendering area of ​​the current scene for 4 consecutive frames, and it is determined that the sampled rendering of each frame is n*n target pixels to be rendered in the 2*2 target area to be rendered. It can be understood that, Fig.11 As shown, in the embodiment of the present application, during rendering, one frame only samples and renders n*n target pixels to be rendered, compared with the original method of sampling and rendering 2*2 target pixels to be rendered in one frame, the sampling and rendering time is effectively shortened to 1 / 4 of the original time, thereby enhancing the frame rate.

[0092] It should be noted that when the frame rate = 180HZ, the interval between frames is 1s / 180HZ, 1s / 180HZ*9 = 50ms, please refer to Fig.12 , an embodiment of the implementation mode of the present application can realize 9-frame superposition, let M=9, that is, determine that the number of pixels to be sampled and rendered in the area to be sampled and rendered is 9, and the number of frames is 9, then as shown in the scene graph, the n*n pixels in the area to be sampled and rendered can be sampled and rendered through 9 consecutive frames to generate a display image.

[0093] See also Fig.13 In order to better implement the display method of the embodiment of the present application, the embodiment of the present application also provides a display device 10. The display device 10 may include a generation module 11 and a display module 12. The generation module 11 is used to perform sampling and rendering on the current scene to generate a display image. The current scene includes multiple areas to be sampled and rendered. There are M matching pixels to be rendered in the areas to be sampled and rendered. The display image is generated by sampling and rendering according to the position of the target pixel to be rendered among the M pixels to be rendered. The positions of the target pixel to be rendered corresponding to the display images of M consecutive frames are different from each other, and M is a positive integer. The display module 12 is used to control the target display pixel of the display screen to work to display the display image. The display screen includes multiple pixel areas. The pixel area includes M display pixels. The target display pixel is the display pixel corresponding to the position of the target pixel to be rendered among the M display pixels.

[0094] In some embodiments, each of the to-be-sampled rendering areas has matching M pixels to be rendered in the pixel array corresponding to the original sampling density, and the original sampling density corresponds to the resolution of the display screen; the sampling rendering includes rasterization and pixel shading, and the generation module 11 is specifically used to determine the target pixels to be rendered corresponding to the current scene according to the frame number of the current scene; the positions of the target pixels to be rendered corresponding to the current scene are rasterized and pixel shaded in sequence to generate the display image, and the rasterization sampling density is 1 / M of the original sampling density.

[0095] In some embodiments, the generating module 11 is further configured to divide the frame number of the current scene by M to determine a remainder; and determine the target pixel to be rendered corresponding to the current scene according to the remainder.

[0096] In some embodiments, the generating module 11 is further configured to perform vertex processing on the current scene to determine a plurality of rendering areas to be sampled.

[0097] In some embodiments, the display device 10 also includes a synchronization module 13, which is used to send a current frame signal to the sampling rendering module and the display control module respectively to synchronize the sampling rendering module and the display control module. The sampling rendering module is configured to generate a display image, and the display control module is configured to control the target display pixels of the display screen to operate by sending a control signal to the display screen, and after the sampling rendering module and the display control module are synchronized, the target pixels to be rendered corresponding to the display image correspond to the target display pixel positions of the display screen.

[0098] The display device is described above from the perspective of functional modules in conjunction with the accompanying drawings. The functional modules can be implemented in hardware form, can be implemented in software form, or can be implemented in combination with hardware and software modules. Specifically, each step of the method implementation in the embodiment of the present application can be completed by the hardware integrated logic circuit in the processor and / or the instructions in software form. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware encoding processor, or can be executed by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps in the above method implementation in combination with its hardware.

[0099] See also Fig.14The display device 100 of the embodiment of the present application includes a display 20, a processor 30, a memory 40 and a computer program, wherein the computer program is stored in the memory 40 and executed by the processor 30, and the computer program includes instructions for executing the display method of any of the above embodiments.

[0100] Optionally, the processor 30 includes a main processor 31 and a graphics processing unit (GPU) 32, and the GPU processor 32 performs sampling and rendering on the current scene to generate a scene image.

[0101] Optionally, the display device of the embodiment of the present application also includes a sampling rendering module 50 and a display control module 60. The sampling rendering module 50 is configured to generate a display image, and the display control module 60 is configured to control the operation of the target display pixels of the display 20 by sending a control signal to the display screen. The frame signal is sent through the main processor 31 to synchronize the sampling rendering module 50 and the display control module 60 so that the polarization angle of the displayed image and the light modulated by the polarization modulator can correspond.

[0102] Optionally, the display device of the embodiment of the present application further includes an optical module, the optical module includes a phase delay plate and a Fresnel lens, or the optical module includes a phase delay plate and a folded optical path component.

[0103] See also Fig.14 and Fig.15 In the case where the current scene is sampled and rendered in the GPU image processor 32 to generate a scene image and the display is controlled by the FPGA, the main processor 31 synchronously sends a frame signal to the GPU and the FPGA. In the sampling and rendering module 50, the GPU processor 32 performs vertex processing according to the application scenario, and generates a display image after rasterization, pixel shading and Framebuffer resampling processing according to the frame signal. At the same time, after receiving the frame signal, the FPGA sends a gate circuit signal to control the target display pixel in the display 20 to work. The GPU sends the generated display image to the display 20 with the selected target display pixel for display. The light of the scene image displayed by the display 20 is superimposed on the human eye through the optical module. The rasterization, pixel shading and Framebuffer resampling processing methods of the GPU processor 32 are continuously switched according to the frame signal, and the target display pixel of the display 20 is synchronously selected for display, and the image is superimposed on the human eye, so that the frame rate of the display image is superimposed to ensure the resolution.

[0104] See also Fig.16The embodiment of the present application also provides a computer-readable storage medium 600 on which a computer program 610 is stored. When the computer program 610 is executed by a processor 620, the steps of the map saving method of any of the above-mentioned embodiments are implemented. For the sake of brevity, they are not repeated here.

[0105] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0106] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0107] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A display method, characterized in that: include: Sampling and rendering are performed on the current scene to generate a display image, wherein the current scene includes a plurality of to-be-sampled rendering areas, wherein the to-be-sampled rendering areas have M matching to-be-rendered pixels, and the display image is generated by sampling and rendering according to the position of a target to-be-rendered pixel among the M to-be-rendered pixels, and the positions of the target to-be-rendered pixels corresponding to the display images of M consecutive frames are different from each other, and M is a positive integer; The target display pixel of the display screen is controlled to work to display the display image, the display screen includes a plurality of pixel areas, the pixel area includes M display pixels, and the target display pixel is a display pixel among the M display pixels corresponding to the position of the target pixel to be rendered.

2. The display method according to claim 1, characterized in that: Each of the to-be-sampled rendering areas has M matching to-be-rendered pixels in a pixel array corresponding to an original sampling density, and the original sampling density corresponds to a resolution of the display screen; The sampling rendering includes rasterization and pixel shading, and the sampling rendering of the current scene to generate a display image includes: Determining the target pixel to be rendered corresponding to the current scene according to the frame sequence number of the current scene; The positions of the target pixels to be rendered corresponding to the current scene are sequentially rasterized and pixel colored to generate the display image, and the sampling density of the rasterization is 1 / M of the original sampling density.

3. The display method according to claim 2, characterized in that: The step of determining the target pixel to be rendered corresponding to the current scene according to the frame sequence number of the current scene includes: Dividing the frame number of the current scene by M to determine a remainder; and The target pixel to be rendered corresponding to the current scene is determined according to the remainder.

4. The display method according to claim 1 or 2, characterized in that: The sampling rendering also includes vertex processing, and the sampling rendering of the current scene to generate a display image also includes: Vertex processing is performed on the current scene to determine a plurality of rendering areas to be sampled.

5. The display method according to claim 1, characterized in that: Also includes: Sending a current frame signal to a sampling rendering module and a display control module respectively to synchronize the sampling rendering module and the display control module, wherein the sampling rendering module is configured to generate the display image, and the display control module is configured to control the target display pixel of the display screen to work by sending a control signal to the display screen; After the sampling rendering module and the display control module are synchronized, the target pixels to be rendered corresponding to the display image correspond to the target display pixel positions on which the display screen operates.

6. The display method according to claim 1, characterized in that: The total display time of the M consecutive frames of the displayed image is less than the preset human eye integration time.

7. The display method according to claim 1, characterized in that: M is equal to 4, the target pixel to be rendered includes 1 pixel to be rendered, and the 4 pixels to be rendered are arranged in a matrix.

8. A display device, characterized in that: The display device comprises: A generation module performs sampling and rendering on a current scene to generate a display image, wherein the current scene includes a plurality of to-be-sampled rendering areas, wherein the to-be-sampled rendering areas have M matching to-be-rendered pixels, and the display image is generated by sampling and rendering according to the position of a target to-be-rendered pixel among the M to-be-rendered pixels, wherein the positions of the target to-be-rendered pixels corresponding to the display images of M consecutive frames are different from each other, and M is a positive integer; The display module is used to control the target display pixel of the display screen to work so as to display the display image. The display screen includes multiple pixel areas, and the pixel area includes M display pixels. The target display pixel is a display pixel among the M display pixels corresponding to the position of the target pixel to be rendered.

9. A display device, characterized in that: include: Processor, memory, display; and A computer program, wherein the computer program is stored in the memory and executed by the processor, and the computer program includes instructions for executing the display method according to any one of claims 1 to 7.

10. The display device according to claim 9, characterized in that It also includes a sampling rendering module and a display control module, wherein the sampling rendering module is configured to generate the display image, and the display control module is configured to control the target display pixel of the display screen to operate by sending a control signal to the display screen.

11. The display device according to claim 9, characterized in that It also includes an optical module, which includes a phase delay plate and a Fresnel lens, or the optical module includes a phase delay plate and a folded optical path component.

12. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the display method according to any one of claims 1 to 7.

Images