Display method, display device, head-mounted display and storage medium

Abstract

The invention discloses a display method, a display device, a head-mounted display and a storage medium. The method is applied to a head-mounted display, the head-mounted display comprises a left-eye display and a right-eye display, and the method comprises the following steps: rasterizing a scene to be rendered to generate a rendering grid; based on a target grid corresponding to the gaze point of the human eye in the rendering grid, performing partition processing on the rendering grid to obtain a plurality of sub-rendering areas, the downsampling multiplying power of the sub-rendering areas formed by the target grid being the minimum value of the downsampling multiplying power of each sub-rendering area; rendering the rendering grid based on the downsampling magnification corresponding to the sub-rendering area to generate a first display image; rendering the rendering grid based on the downsampling magnification and the preset pixel offset corresponding to the sub-rendering area to generate a second display image; the first display image is displayed through one of the left-eye display and the right-eye display, and the second display image is displayed through the other one, so that the resolution is improved.

Description

Technical Field

[0001] The present application belongs to the field of virtual reality technology, and in particular, relates to a display method, a display device, a head-mounted display, and a non-transitory computer-readable storage medium. Background Art

[0002] Currently, a head mounted display (HMD) processes images and projects the processed images to human eyes, thereby providing an immersive experience for users.

[0003] However, due to the size of the headset, its image processing performance has a bottleneck and the computing power of image rendering has an upper limit. Therefore, for high-resolution display images, the rendering speed is difficult to match the display refresh rate. Conversely, if you want to increase the rendering speed to match the display refresh rate, you need to sacrifice some resolution. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a display method, a display device, a head-mounted display, and a non-transient computer-readable storage medium, which can ensure the resolution of the displayed image when the rendering speed matches the refresh rate of the display screen.

[0005] The display method of the present application is applied to a head-mounted display, the head-mounted display comprising a left-eye display and a right-eye display, the method comprising: rasterizing a scene to be rendered to generate a rendering grid; partitioning the rendering grid based on a target grid corresponding to a gaze point of a human eye in the rendering grid to obtain a plurality of sub-rendering areas, wherein a downsampling magnification of the sub-rendering area formed by the target grid is a minimum value of the downsampling magnification of each sub-rendering area; rendering the rendering grid based on the downsampling magnification corresponding to the sub-rendering area to generate a first display image; rendering the rendering grid based on the downsampling magnification corresponding to the sub-rendering area and a preset pixel offset to generate a second display image; displaying the first display image through one of the left-eye display and the right-eye display, and displaying the second display image through the other.

[0006] In some embodiments, the downsampling ratio of the sub-rendering area during rendering is proportional to the distance between the sub-rendering area and the target grid.

[0007] In some implementations, the preset pixel offset of the sub-rendering area is proportional to the downsampling ratio when rendering the sub-rendering area.

[0008] In some implementations, the preset pixel offset corresponding to the sub-rendering area is equal to 1 / 2 of the downsampling ratio of the sub-rendering area.

[0009] In some embodiments, the preset pixel offset of the sub-rendering area including the target grid is equal to 0.

[0010] In some embodiments, the rendering grid is rendered based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image, including: determining an offset grid corresponding to any grid of the sub-rendering area based on the preset pixel offset; rendering any grid based on the downsampling ratio corresponding to the sub-rendering area and image information corresponding to the offset grid to generate the second display image.

[0011] In some embodiments, the offset direction corresponding to the preset pixel offset is at least one of a horizontal direction, a vertical direction, and a diagonal direction.

[0012] The display device of the present application is applied to a head-mounted display, the head-mounted display includes a left-eye display and a right-eye display, and the display device includes a first generation module, a processing module, a second generation module, a third generation module and a display module. The first generation module is used to rasterize the scene to be rendered to generate a rendering grid; the processing module is used to partition the rendering grid based on the target grid corresponding to the gaze point of the human eye in the rendering grid to obtain multiple sub-rendering areas, and the downsampling magnification of the sub-rendering area composed of the target grid is the minimum value of the downsampling magnification of each sub-rendering area; the second generation module is used to render the rendering grid based on the downsampling magnification corresponding to the sub-rendering area to generate a first display image; the third generation module is used to render the rendering grid based on the downsampling magnification corresponding to the sub-rendering area and a preset pixel offset to generate a second display image; the display module is used to display the first display image through one of the left-eye display and the right-eye display, and display the second display image through the other.

[0013] The head-mounted display of the present application includes a left-eye display, a right-eye display, a memory and a processor; the memory stores a computer program, and when the processor executes the program, the display method described in any of the above embodiments is implemented.

[0014] The non-transitory computer-readable storage medium of the present application stores a computer program thereon, and when the computer program is executed by a processor, the display method as described in any of the above embodiments is implemented.

[0015] The display method, display device, head-mounted display, and non-transitory computer-readable storage medium provided in the embodiments of the present application render a rendering grid based on the downsampling ratio corresponding to each sub-rendering area to generate a first display image, and render the rendering grid based on the downsampling ratio corresponding to the sub-rendering area and a preset pixel offset to generate a second display image, thereby improving the rendering speed of the first display image and the second display image; and then by displaying the first display image on one of the left-eye display and the right-eye display of the head-mounted display and displaying the second display image on the other, the first display image and the second display image are shifted and superimposed in the human eye, and the human eye can perceive more scene details, thereby improving the resolution and avoiding the occurrence of color mutations and the like that affect the user experience, that is, the image resolution is guaranteed while improving the image rendering speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0017] Figure 1 It is an application scenario diagram of the display method provided in the embodiment of the present application;

[0018] Figure 2 It is a flowchart of a display method provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a scene of a display method provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a scene of a display method provided in an embodiment of the present application;

[0021] Figure 5 It is a flowchart of a display method provided in an embodiment of the present application;

[0022] Figure 6 It is a flowchart of a display method provided in an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of a module of a display device provided in an embodiment of the present application;

[0024] Figure 8 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0026] For ease of understanding, the technical background and application scenarios of this application are first introduced below:

[0027] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR includes Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR).

[0028] Augmented Reality (AR): also known as augmented reality, is a technology that cleverly integrates virtual information with the real world. It widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and other technical means to simulate computer-generated text, images, three-dimensional models, music, video and other virtual information, and apply it to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0029] Virtual Reality (VR): also known as virtual reality or spiritual realm technology. Virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments in three-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other high-tech technologies. With the help of the graphics processing unit (GPU) in the VR device, the image in the current scene is processed to produce a realistic three-dimensional visual, tactile, olfactory and other sensory experiences in the virtual world, so that people in the virtual world have an immersive feeling.

[0030] Mixed Reality (MR) technology refers to a new visualization environment created by merging the real and virtual worlds, in which physical and digital objects coexist and can interact with the real world in real time and obtain information in a timely manner.

[0031] In order to more vividly illustrate the various technical solutions in the embodiments of the present disclosure, the image processing method of the present application is introduced by taking virtual reality (VR) and head-mounted display as an example of VR head display used in VR. It can be understood that for XR technology, the principle of implementing the image processing method of the present application is basically similar and will not be repeated here. It should be emphasized that this is only an example and is not a specific limitation on the scope of application of the embodiments of the present disclosure.

[0032] See also Figure 1 , Figure 1 1 is an application scenario diagram of an image processing method provided in an embodiment of the present application. The application scenario provided in the present application includes a head-mounted display 100 (such as a VR head-mounted display), the head-mounted display 100 includes a body 10, a left-eye display 20 and a right-eye display 30, the left-eye display 20 displays a left-eye image, and the right-eye display 30 displays a right-eye image.

[0033] In one embodiment, the head mounted display 100 may be a video projection head mounted display, a holographic head mounted display, an AR head mounted display, a VR head mounted display, etc.

[0034] In one embodiment, the head mounted display 100 may include at least one image acquisition device 40 (such as a visible light camera, an infrared camera, a structured light camera, etc.) for acquiring a two-dimensional or three-dimensional image of an actual scene.

[0035] In one embodiment, the head mounted display 100 includes at least one of an image signal processing unit (ISP) 50 and a graphics processing unit (GPU) 60. The ISP 50 can be used to process the image signal of the image acquisition device 40, such as performing color correction, noise reduction, gamma correction, etc., to improve the image quality.

[0036] The GPU 60 can render the virtual scene, such as vertex processing, rasterization, pixel shading, etc., to obtain a virtual scene image.

[0037] Among them, vertex processing is to convert the coordinates of the three-dimensional space corresponding to the vertices of the 3D graphics into the two-dimensional space of the display through linear algebra calculations, and at the same time calculate the color of the vertex for subsequent pixel color interpolation operations.

[0038] Among them, the rasterization process is to input the primitive information generated during vertex processing into the rasterization stage, first assemble the primitives, and then determine the fragments corresponding to the screen pixels covered by the primitives through triangle traversal. After the rasterization stage, the primitives will be divided into basic units of pixel size, which are called fragments. Fragments are more like data representations of pixels. The final pixels are generated by the information in the fragments, and then the fragments are colored by the fragment shader. When operating fragment by fragment, the fragments undergo a series of test processes. The tested fragments will be converted into pixels and finally presented in the frame buffer area.

[0039] The pixel coloring process may be to calculate the color information corresponding to each pixel, that is, the pixel value.

[0040] In one embodiment, the head-mounted display 400 can also fuse the virtual scene image obtained after rendering by the GPU 50 with the image processed by the ISP 60 to obtain a fused image, and then display the fused image through the left eye display and the right eye display, thereby achieving an augmented reality visual effect.

[0041] Based on the introduction of the above-mentioned related scenarios, the embodiment of the present application provides a display method, which is described in detail below:

[0042] See also Figure 2 , Figure 2 011, 012, 013, 014, and 015. The head-mounted display is a VR device, and the method is implemented as follows.

[0043] Step 011: Rasterize the scene to be rendered to generate a rendering mesh.

[0044] Among them, the scene to be rendered can be a scene or image that can be displayed on the left eye display or the right eye display after rendering. The scene to be rendered can be an actual scene or a virtual scene, etc.; the rendering grid is a grid formed by fragments after the scene to be rendered is rasterized.

[0045] Specifically, a first visual sensor (i.e., an image acquisition device) can be set on the head mounted display, and the position and direction of the first visual sensor can simulate the user's eyes. According to the position and direction of the first visual sensor, the scene to be rendered presented to the user in the current scene when the user uses the VR device is determined, and the scene to be rendered is rasterized to generate a rendering grid.

[0046] For example, the virtual scene can be a three-dimensional (3D) scene or a two-dimensional scene. Taking the virtual scene as a 3D scene as an example, the scene to be rendered in the current scene is determined according to the first visual sensor set to simulate the user's eyes, and the view of the scene to be rendered is transformed through the GPU of the head-mounted display. The three-dimensional coordinates of the 3D scene graph are converted into two-dimensional coordinates, and then vertex processing and rasterization are performed to obtain a rendering mesh of the scene to be rendered.

[0047] like Figure 3 As shown, after rasterizing the rendered scene, we can get the following Figure 3 The rendering grid shown, one grid corresponds to one pixel.

[0048] Step 012: Based on the target grid corresponding to the gaze point of the human eye in the rendering grid, the rendering grid is partitioned to obtain multiple sub-rendering areas, and the downsampling ratio of the sub-rendering area composed of the target grid is the minimum value of the downsampling ratios of each sub-rendering area.

[0049] The gaze point may be a point or area that the user's eyes gaze at when using the head-mounted display.

[0050] The target grid may be a grid corresponding to the gaze point of the human eye in the rendering grid, or a grid corresponding to the gaze point of the human eye in the rendering grid and grids around the grid. The target grid may be one or more, such as a 1*1 grid, a 2*2 grid, a 3*3 grid, etc.

[0051] The downsampling ratio can be 1*1, 2*2, 3*3, etc., which can be used to represent the degree to which the image resolution is reduced when rendering the rendering area. For example, if the rendering grid resolution is 100*100, after downsampling and rendering at a downsampling ratio of 1*1, a 100*100 image will be obtained; for another example, if the rendering grid resolution is 100*100, after downsampling and rendering at a downsampling ratio of 2*2, a 50*50 image will be obtained.

[0052] The downsampling ratio corresponds to the shading rate of pixel shading during the rendering process, such as the shading rate is equal to the downsampling ratio. The shading rate refers to the number of grids in a single shading area, and each grid in a single shading area is shaded based on the same color information. When shading pixels, if the shading rate is 1*1, the number of grids shaded at a single time is 1*1, that is, one grid is shaded each time; if the shading rate is 2*2, the number of grids shaded at a single time is 2*2, that is, 2*2 grids are shaded each time.

[0053] The sub-rendering areas obtained after the partitioning process may be regular shapes, such as rectangular rings, circular rings, etc., or irregular shapes, etc.

[0054] Specifically, an infrared camera can be set up to capture an image of a person's eyeball, and then the center of the pupil of the human eye and other feature points can be located by detecting infrared reflection, etc. Based on the located center of the pupil of the human eye and other feature points, the gaze direction of the human eye can be calculated, and according to the gaze direction of the human eye, the user's gaze point position can be calculated by, for example, a geometric projection method. After determining the gaze point position, the gaze point position is mapped to the rendering grid generated after rasterization of the scene to be rendered to determine the target grid.

[0055] Then, according to the target grid corresponding to the gaze point of the human eye in the rendering grid, when the rendering grid generated by the rendering scene is partitioned, the target grid can be taken as the center and used as a sub-rendering area, and the distance values ​​of each grid in the rendering grid and the target grid are calculated. Then, by setting multiple distance ranges, the distance values ​​of each grid and the target grid (such as the number of grids that differ between each grid and the target grid in the horizontal direction or the vertical direction as the distance value of each grid and the target grid) are located in the same distance range as a sub-rendering area, thereby partitioning to obtain multiple sub-rendering areas.

[0056] For example, see Figure 4 , with the target grid b in the rendering grid as the center, the target grid is taken as a sub-rendering area 1, and then by setting the first distance range (such as [0, 2], the unit is grid, that is, the grids within the range of 0 grids to 2 grids from the target grid b are taken as the sub-rendering area 2 corresponding to the first distance range), and then by setting the second distance range (such as [2, 5], the unit is grid, that is, the grids within the range of 2 grids to 5 grids from the target grid b are taken as the sub-rendering area 3 corresponding to the second distance range).

[0057] It should be noted that the downsampling ratio of the sub-rendering area composed of the target grid is the minimum of the downsampling ratios of each sub-rendering area. That is, in the image generated after rendering, the image portion corresponding to the sub-rendering area composed of the target grid has the highest resolution. Since the resolution of the area where the gaze point is located is the highest resolution, it can not only save the computing power of image transmission and calculation, but also ensure the user experience.

[0058] For example, see again Figure 4 , the downsampling ratio of the sub-rendering area 1 where the target mesh b is located is 1*1, which is smaller than the downsampling ratio of the sub-rendering area 2 (2*2), and smaller than the downsampling ratio of the sub-rendering area 3 (4*4). That is, the sub-rendering area 1 containing the target mesh b has the highest resolution in the image generated after rendering.

[0059] Optionally, the downsampling ratio of the sub-rendering area containing the target mesh is 1*1.

[0060] For example, see Figure 4 , a downsampling ratio of 1*1 is used to sample the sub-rendering area 1 containing the target mesh, a downsampling ratio of 2*2 is used for the sub-rendering area 2, and a downsampling ratio of 4*4 is used for the sub-rendering area 3, so as to ensure that the sub-rendering area 1 containing the target mesh b has the lowest downsampling ratio during sampling, that is, in the generated image, the resolution of the user's gaze point area is the highest, thereby improving the user's visual experience.

[0061] Optionally, the downsampling ratio when rendering the sub-rendering area is proportional to the distance between the sub-rendering area and the target grid.

[0062] Specifically, the downsampling ratio of the sub-rendering area during rendering increases as the distance between the sub-rendering area and the target grid increases. The farther the sub-rendering area is from the target grid, the higher the downsampling ratio is during rendering and the lower the resolution is.

[0063] For example, see again Figure 4 , the distance between sub-rendering area 3 and the target grid (sub-rendering area 1) is greater than the distance between sub-rendering area 2 and the target grid (sub-rendering area 1). When rendering, the downsampling ratio of sub-rendering area 3 (such as Figure 4 In the example, the downsampling ratio of sub-rendering area 3 is 3*3) which is greater than the downsampling ratio of sub-rendering area 2 (e.g. Figure 4 In the example, the downsampling ratio of sub-rendering area 2 is 2*2).

[0064] Step 013: Rendering the rendering grid based on the downsampling ratio corresponding to the sub-rendering area to generate a first display image.

[0065] Specifically, when determining the downsampling ratio corresponding to each sub-rendering area, the shading rate of each sub-rendering area can be determined based on the downsampling ratio corresponding to each sub-rendering area. The shading rate is the same as the downsampling rate. According to the shading rate of each sub-rendering area, the grid of each sub-rendering area is shaded to obtain the pixel value of each sub-rendering area. For example, when the downsampling ratio is 2*2, the shading rate is 2*2, that is, each time a 2*2 grid is shaded, and the pixel value of each sub-rendering area is calculated in turn, and the first display image can be determined.

[0066] Step 014: Rendering the rendering grid based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image.

[0067] The preset pixel offset can be 1, 2, 3, 4, 5, etc., and the unit is a grid. If the preset pixel offset is 1, the pixel offset is 1 grid.

[0068] Specifically, when rendering each grid of the rendering grid to obtain corresponding pixels, a preset pixel offset may also be considered. For example, based on the image data information of a grid that has a preset pixel offset distance from the current grid to be rendered, the current grid to be rendered is sampled and rendered to obtain pixels corresponding to the current grid to be rendered.

[0069] For example, see Figure 3 , taking the preset pixel offset as 1 grid along the diagonal direction of the grid (i.e., one grid offset in both the horizontal and vertical directions) as an example, after the rendered scene is rasterized, the following is obtained Figure 3 For the 10*10 rendering grid shown, when sampling and rendering the rendering grid (m, n) at the mth row and nth column, the image information used is the image information of the rendering grid (m+1, n+1) at the m+1th row and n+1th column. For example, when sampling and rendering the rendering grid (1, 1) at the 1st row and 1st column, the image information used is the image information of the rendering grid (2, 2) at the 2nd row and 2nd column, and when sampling and rendering the rendering grid (1, 2) at the 1st row and 2nd column, the image information used is the image information of the rendering grid (2, 3) at the 2nd row and 3rd column. In this way, the rendering grids generated after rasterization of the rendered scene can be sequentially sampled and rendered with preset pixel offsets.

[0070] When generating the second display image, in addition to considering the downsampling ratio corresponding to each sub-rendering area, the preset pixel offset of each sub-rendering area also needs to be considered. Based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area, the corresponding sub-rendering area is rendered to obtain pixel values ​​different from those of each sub-rendering area without the preset pixel offset, and the pixel values ​​of each sub-rendering area after the offset can determine the second display image.

[0071] Optionally, the preset pixel offset of the sub-rendering area is proportional to the downsampling ratio when the sub-rendering area is rendered.

[0072] Specifically, the larger the downsampling ratio of the sub-rendering area, the larger the size of a single pixel after downsampling. For example, when the downsampling ratio is 2*2, the shading rate is 2*2, that is, each time a 2*2 grid is shaded, the 4 pixels corresponding to the 2*2 grid can be used as a large pixel. When pixels are shifted and superimposed, in order to ensure the superposition effect and avoid the problem of poor imaging quality caused by superimposing scenes with too large differences, the pixel offset distance is generally less than the distance of one pixel. Therefore, the preset pixel offset can be proportional to the downsampling ratio of the sub-rendering area, that is, the larger the downsampling ratio of the sub-rendering area, the larger the preset pixel offset corresponding to the sub-rendering area.

[0073] Optionally, the preset pixel offset corresponding to the sub-rendering area is equal to 1 / 2 of the downsampling ratio of the sub-rendering area to achieve a half-pixel offset.

[0074] Specifically, the preset pixel offset corresponding to the sub-rendering area is equal to 1 / 2 of the downsampling ratio of the sub-rendering area, specifically 1 / 2 of the downsampling ratio in the horizontal direction or the vertical direction. For example, taking the pixel offset in the diagonal direction as an example, when the downsampling ratio of the sub-rendering area is 2P*2P, when generating the second display image, the preset pixel offset of the sub-rendering area is P, that is, when sampling, the sub-rendering area is offset by P grids in the horizontal direction and P grids in the vertical direction, thereby achieving half-pixel offset in the horizontal direction and the vertical direction, and the effect of shift superposition is better after the half-pixel offset.

[0075] For another example, taking pixel offset in the horizontal direction, when the downsampling ratio of the sub-rendering area is 2P*2P, the preset pixel offset corresponding to the sub-rendering area is P, that is, during sampling, the sub-rendering area is offset by P grids in the horizontal direction, thereby achieving a half-pixel offset in the horizontal direction. After the half-pixel offset, the effect of shift superposition is better.

[0076] Optionally, the preset pixel offset of the sub-rendering area containing the target mesh is equal to 0.

[0077] Specifically, when the human eye views the first display image and the second display image, the sub-rendering area where the gaze point of the first display image is located and the sub-rendering area where the gaze point of the second display image is located need to be aligned in the human eye to ensure that the image contents seen by the left eye and the right eye of the person can correspond or overlap visually. That is, the preset pixel offset of the sub-rendering area where the target grid is located needs to be set to 0, and no pixel offset is performed on the area where the gaze point is located. If the sub-rendering area where the gaze point of the first display image is located and the sub-rendering area where the gaze point of the second display image is located are not aligned, the head-mounted display may not provide the user with correct scene perception, and may also affect the user's experience.

[0078] In addition, since the downsampling rate of the sub-rendering area where the gaze point is located is low and the resolution is already high, the preset pixel offset of the sub-rendering area where the target grid is located is 0, which also avoids the problem of image quality degradation caused by pixel offset and ensures the display effect of the gaze point.

[0079] Optionally, the offset direction corresponding to the preset pixel offset is at least one of a horizontal direction, a vertical direction and a diagonal direction.

[0080] Specifically, the horizontal direction can be Figure 3 The row direction of the rendering grid in is Figure 3 The column direction of the rendering grid in , the diagonal direction can be Figure 3 The diagonal direction of the grid.

[0081] Taking the preset pixel offset direction as horizontal to the right and the pixel offset as 1 as an example, when sampling and rendering the rendering grid (m, n) of the mth row and nth column, the image information used is the image information of the rendering grid (m, n+1) of the mth row and n+1th column; taking the preset pixel offset direction as vertically downward and the pixel offset as 2 as an example, when sampling and rendering the rendering grid (m, n) of the mth row and nth column, the image information used is the image information of the rendering grid (m, n) of the m+2th row and nth column; taking the preset pixel offset direction as along the diagonal direction from the upper left to the lower right and the pixel offset as 1 as an example, when sampling and rendering the rendering grid (m, n) of the mth row and nth column, the image information used is the image information of the rendering grid (m+1, n+1) of the m+1th row and n+1th column.

[0082] Optionally, the preset pixel offset may be a combination of different offset sizes in the horizontal direction, the vertical direction and the diagonal direction, so as to achieve offsets at various angles.

[0083] Due to the presence of a preset pixel offset, for the same scene to be rendered, a first display image and a second display image with different scene details can be generated respectively. Compared with the absence of a preset pixel offset, that is, the first display image and the second display image are basically consistent, the rendered first display image and the second display image can respectively contain different scene details of the same scene.

[0084] Step 015: Display the first display image on one of the left-eye display and the right-eye display, and display the second display image on the other.

[0085] Specifically, the first display image is displayed by one of the left eye display and the right eye display of the head mounted display, and the second display image is displayed by the other. For example, the left eye display of the head mounted display displays the first display image, and the right eye display displays the second display image; or, the left eye display of the head mounted display displays the second display image, and the right eye display displays the first display image.

[0086] The sensitivity of the human visual system to resolution decreases sharply as the gaze center of the human eye moves toward the periphery of the field of view. Existing head-mounted displays take advantage of this characteristic of the human eye when rendering, using gaze point rendering technology to render the image, thereby saving computing power and bandwidth.

[0087] Gaze point rendering technology is a graphics computing technology. During sampling and rendering, the area where the human eye is looking at is taken as the center, and the sampling rate is gradually increased to the surrounding areas. That is, the rendering resources are concentrated in the area where the human eye is looking at, and the area where the human eye is looking at is rendered with high resolution, while other areas are rendered with lower resolution, thereby reducing computing power and ensuring the smoothness of the displayed image.

[0088] However, when rendering the current scene based on gaze point rendering technology, the resolution of the area farther away from the gaze point is reduced more seriously. Therefore, when the gaze point changes, the area far away from the gaze point may experience color mutation due to large-scale downsampling, affecting the user experience.

[0089] However, the present application displays the first display image on one of the left-eye display and the right-eye display of the head-mounted display and displays the second display image on the other. When the user views the picture of the scene to be rendered through the head-mounted display, since there is a preset pixel offset when the first display image and the second display image are rendered, the obtained first display image and the second display image respectively contain different scene details of the same scene to be rendered. The first display image and the second display image can be shifted and superimposed in the human eye, so that the human eye can perceive more scene details, achieving an effect basically the same as the resolution improvement achieved by image shifting and superposition, and basically does not affect the display frame rate, thereby improving the display effect.

[0090] That is to say, the present application renders the rendering grid based on the downsampling ratio corresponding to each sub-rendering area to generate a first display image, and renders the rendering grid based on the downsampling ratio corresponding to the sub-rendering area and the preset pixel offset to generate a second display image, thereby improving the rendering speed of the first display image and the second display image; and then by displaying the first display image on one of the left-eye display and the right-eye display of the head-mounted display and displaying the second display image on the other, the first display image and the second display image are shifted and superimposed in the human eye, and the human eye can perceive more scene details, thereby improving the resolution and avoiding color mutations that affect the user experience, that is, ensuring the image resolution while improving the image rendering speed.

[0091] It should be noted that the first display image and the second display image can be generated simultaneously or sequentially. The first display image and the second display image can be displayed simultaneously or sequentially. For the convenience of description, the first display image is displayed in the left eye display and the second display image is displayed in the right eye display as an example for description.

[0092] (1) The first display image and the second display image are generated simultaneously, and the first display image is displayed in the left eye display and the second display image is displayed in the right eye display simultaneously, that is, step 013 and step 014 are executed simultaneously, and step 015 displays the first display image and the second display image simultaneously.

[0093] For example, at time T1, the first display image and the second display image are generated simultaneously, and at time T2, the first display image is displayed on the left-eye display, and the second display image is displayed on the right-eye display.

[0094] (2) The first display image and the second display image are generated simultaneously, and there is a time difference between the first display image and the second display image when they are displayed, wherein the time difference during display is less than the integration time, that is, step 013 and step 014 are executed simultaneously, and step 015 is executed in a time-sharing manner to display the first display image and the second display image.

[0095] The human eye has a persistence of vision characteristic, that is, it takes a certain integration time for an image to form on the retina, and the image displayed during the integration time will be superimposed on the retina. Based on the persistence of vision characteristic of the human eye, the human eye obtains the superimposed image within the integration time.

[0096] Therefore, when the first display image and the second display image are generated simultaneously and the time difference between the display of the first display image and the second display image is less than the integration time, the first display image and the second display image can be superimposed on the human eye, and the human eye can perceive more scene details, thereby achieving the effect of improving the resolution.

[0097] For example, taking the example that the first display image and the second display image are generated simultaneously at time T3, at time T4, the first display image is displayed in the left eye display, and at time T4+△T1, the second display image is displayed in the right eye display, and △T1 is less than the integration time of the human eye.

[0098] (3) The first display image and the second display image are generated successively, and the first display image and the second display image are displayed simultaneously, wherein the time difference between the generation of the first display image and the second display image is less than the integration time, that is, step 013 and step 014 are executed in time-sharing manner, and step 015 displays the first display image and the second display image simultaneously.

[0099] Similarly, taking the example that the first display image is generated at time T5 and the second display image is generated at time T5+△T2, at time T6, the first display image is displayed in the left eye display and the second display image is displayed in the right eye display, wherein △T2 is less than the integration time of the human eye. Based on the visual persistence characteristics of the human eye, the first display image and the second display image can be superimposed in the human eye to achieve improved resolution.

[0100] (4) The first display image and the second display image are generated successively, and the first display image and the second display image are displayed successively, wherein the sum of the time difference during generation and the time difference during display is less than the integration time of the human eye, that is, step 013 and step 014 are performed in time-sharing, and step 015 displays the first display image and displays the second display image in time-sharing.

[0101] Similarly, taking the example that the first display image is generated at time T7 and the second display image is generated at time T7+△T3, at time T8, the first display image is displayed in the left eye display, and at time T8+△T4, the second display image is displayed in the right eye display, where △T3+△T4 is less than the integration time of the human eye. Based on the visual persistence characteristics of the human eye, the first display image and the second display image can be superimposed in the human eye to achieve improved resolution.

[0102] See also Figure 5 In some embodiments, step 014: rendering the rendering grid based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image includes:

[0103] Step 0141: Based on a preset pixel offset, determine an offset grid corresponding to any grid in the sub-rendering area;

[0104] Step 0142: Render any grid based on the downsampling ratio corresponding to the sub-rendering area and the image information corresponding to the offset grid to generate a second display image.

[0105] For details, please continue to see Figure 4 For example, the downsampling ratio of sub-rendering area 1 is 1*1, the downsampling ratio of sub-rendering area 2 is 2*2, and the downsampling ratio of sub-rendering area 3 is 4*4. That is, the shading rate of sub-rendering area 1 is 1*1, the shading rate of sub-rendering area 2 is 2*2, and the shading rate of sub-rendering area 3 is 4*4.

[0106] When generating the first display image, the preset pixel offset is not considered, and for sub-rendering area 1, 1*1 grids are colored each time; for sub-rendering area 2, 2*2 grids are colored each time; and for sub-rendering area 3, 3*3 grids are colored each time. In this way, the pixel values ​​corresponding to each grid of each sub-rendering area can be obtained, thereby determining the first display image.

[0107] When generating the second display image, it is necessary to consider the preset pixel offset corresponding to each sub-rendering area. If the grid currently being rendered is located in the sub-rendering area 1 in the rendering grid, such as the grid currently being rendered is the target grid b, no pixel offset is performed, and sampling and rendering are performed according to the image information of the target grid b. If the grid being rendered (such as the grid with coordinates (4, 5) in the rendering grid, that is, the grid in the 4th row and the 5th column) is located in the sub-rendering area 2 in the rendering grid, based on the preset pixel offset 1*1, it is determined that the offset grid corresponding to the grid with coordinates (4, 5) is the grid with coordinates (5, 6). Based on the downsampling ratio 2*2 of the sub-rendering area 2 and the image information corresponding to the offset grid with coordinates (5, 6), Perform pixel coloring on the grid with coordinates (4, 5); if the grid to be rendered (such as the grid with coordinates (1, 1) in the rendering grid, that is, the grid in the first row and the first column) is located in the sub-rendering area 3 in the rendering grid, based on the preset pixel offset 2*2, determine that the offset grid corresponding to the grid with coordinates (1, 1) is the grid with coordinates (3, 3), and based on the downsampling ratio 4*4 and the image information of the offset grid with coordinates (3, 3), perform pixel coloring on the grid with coordinates (1, 1), determine the offset grid corresponding to each grid in each sub-rendering area in turn, and then render each grid based on the downsampling ratio corresponding to each sub-rendering area and the image information corresponding to the offset grid, thereby generating a second display image.

[0108] When generating the second display image, the current grid can also be rendered based on the image information of the current grid and the image information of the grid after the preset pixel offset. If the grid currently being rendered is located in sub-rendering area 1 in the rendering grid, if the grid currently being rendered is the target grid b, no pixel offset is performed, and sampling rendering is performed according to the image information of the target grid b; if the grid being rendered (such as the grid with coordinates (4, 5) in the rendering grid, that is, the grid in the 4th row and the 5th column) is located in sub-rendering area 2 in the rendering grid, then based on the downsampling ratio of 2*2 and the preset pixel offset of 1*1, the grid with coordinates (4, 5) can be sampled and rendered based on the image information of the grid with coordinates (5, 6) and the image information of the grid with coordinates (4, 5). 5); if the rendered grid (such as the grid with coordinates (1, 1) in the rendering grid, that is, the grid in the first row and the first column) is located in the sub-rendering area 3 in the rendering grid, then based on the downsampling ratio 4*4 and the preset pixel offset 2*2, according to the image information of the grid with coordinates (3, 3) combined with the image information of the grid with coordinates (1, 1), the grid with coordinates (1, 1) is pixel colored, and each grid in each sub-rendering area is rendered with the corresponding downsampling ratio and the corresponding preset pixel offset in turn, so as to generate a second display image.

[0109] When using a head-mounted device to view the first display image and the second display image, since each sub-rendering area in the second display image, except for the sub-rendering area where the gaze point area is located, is sampled with a corresponding preset pixel offset during sampling, the first display image and the second display image respectively contain different scene details of the scene to be rendered. The first display image and the second display image will be shifted and superimposed in the human eye, so that the human eye can perceive more scene details, achieving an effect that is basically the same as the resolution improvement achieved by image shifting and superposition, and basically does not affect the display frame rate, thereby achieving an improvement in the display effect.

[0110] To facilitate understanding of the display method, the following is an overall introduction to the various steps of image generation and display in the display method. Figure 6 , Figure 6 This is a complete flow diagram of generating and displaying a first display image and a second display image in a display method provided in an embodiment of the present application, wherein, taking the first display image being displayed on a left-eye display and the second display image being displayed on a right-eye display as an example, the display method includes:

[0111] Step 021: partitioning the rendering grid based on the gaze point position to generate multiple sub-rendering areas;

[0112] Step 022: Rendering the corresponding sub-rendering area based on the downsampling ratio corresponding to the sub-rendering area to generate a first display image;

[0113] Step 023: Displaying a first display image through the left-eye display;

[0114] Step 024: Rendering the corresponding sub-rendering area based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image;

[0115] Step 025: Display the second display image through the right eye display.

[0116] Among them, for the specific description of step 021, please refer to the description of step 011 and step 012, for the specific description of step 022, please refer to the description of step 013, for the specific description of step 023, please refer to the description of step 015, for the specific description of step 024, please refer to the description of step 014, step 0141 and step 0142, for the specific description of step 025, please refer to the description of step 015, and for the sake of brevity, they will not be repeated here.

[0117] According to the method described in the above embodiment, the present application embodiment also provides a display device 300 for executing the steps in the above display method. Figure 7 , Figure 7 3 is a schematic diagram of the structure of a display device 300 provided in an embodiment of the present application. The display device 300 includes a first generation module 301, a processing module 302, a second generation module 303, a third generation module 304 and a display module 305, wherein:

[0118] The first generating module 301 is used to rasterize the scene to be rendered to generate a rendering grid;

[0119] The processing module 302 is used to partition the rendering grid based on the target grid corresponding to the gaze point of the human eye in the rendering grid to obtain multiple sub-rendering areas, and the downsampling magnification of the sub-rendering area including the target grid is the minimum value of the downsampling magnifications of each sub-rendering area;

[0120] The second generating module 303 is used to render the rendering grid based on the downsampling ratio corresponding to the sub-rendering area to generate a first display image;

[0121] The third generating module 304 is used to render the rendering grid based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image;

[0122] The display module 305 is configured to display a first display image through one of a left-eye display and a right-eye display, and display a second display image through the other one of the left-eye display and the right-eye display.

[0123] It should be noted that the specific details of each module unit in the above-mentioned display device 300 have been described in detail in the embodiment of the above-mentioned display method, and will not be repeated here.

[0124] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0125] In some embodiments, the display device 300 in the embodiment of the present application can be implemented in hardware, such as an electronic device, or a component in an electronic device, such as an integrated circuit or a chip; the display device 300 can also be implemented in software, such as as an application installed in an electronic device.

[0126] In some embodiments, see Figure 1 , Figure 1 1 is a schematic diagram of the structure of the head mounted display provided in the embodiment of the present application. The head mounted display 100 includes a processor 70, a memory 80, a left eye display 20 and a right eye display 30. The memory 80 stores a computer program 81 that can be run on the processor 70. When the program 81 is executed by the processor 70, each process of the embodiment of the above display method is implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0127] The present application also provides a non-transitory computer-readable storage medium. Figure 8 The non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the various processes of the above-mentioned display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0128] The processor may be a processor in the electronic device in the above embodiment. The computer readable storage medium may be a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0129] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A display method, characterized in that: Applied to a head mounted display, the head mounted display comprises a left eye display and a right eye display, the method comprising: Rasterize the scene to be rendered to generate a rendering mesh; Based on a target grid corresponding to a gaze point of a human eye in the rendering grid, partitioning the rendering grid to obtain a plurality of sub-rendering areas, wherein a downsampling magnification of the sub-rendering area formed by the target grid is a minimum value of the downsampling magnifications of all the sub-rendering areas; Rendering the rendering grid based on the downsampling ratio corresponding to the sub-rendering area to generate a first display image; Rendering the rendering grid based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image; The first display image is displayed through one of the left-eye display and the right-eye display, and the second display image is displayed through the other.

2. The display method according to claim 1, characterized in that: The downsampling ratio of the sub-rendering area during rendering is proportional to the distance between the sub-rendering area and the target grid.

3. The display method according to claim 1 or 2, characterized in that: The preset pixel offset of the sub-rendering area is proportional to the downsampling ratio when the sub-rendering area is rendered.

4. The display method according to claim 1 or 3, characterized in that: The preset pixel offset corresponding to the sub-rendering area is equal to 1 / 2 of the downsampling ratio of the sub-rendering area.

5. The display method according to claim 3 or 4, characterized in that: The preset pixel offset of the sub-rendering area containing the target grid is equal to 0.

6. The display method according to claim 1, characterized in that: The rendering grid is rendered based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image, including: Based on the preset pixel offset, determining an offset grid corresponding to any grid in the sub-rendering area; Based on the downsampling ratio corresponding to the sub-rendering area and the image information corresponding to the offset grid, the any one grid is rendered to generate the second display image.

7. The display method according to any one of claims 1 to 6, characterized in that: The offset direction corresponding to the preset pixel offset is at least one of a horizontal direction, a vertical direction and a diagonal direction.

8. A display device, characterized in that: Applied to a head mounted display, the head mounted display comprises a left eye display and a right eye display, the device comprises: A first generation module, used for rasterizing a scene to be rendered to generate a rendering grid; A processing module, configured to partition the rendering grid based on a target grid corresponding to a gaze point of a human eye in the rendering grid to obtain a plurality of sub-rendering areas, wherein a downsampling magnification of the sub-rendering area formed by the target grid is a minimum value of the downsampling magnifications of all the sub-rendering areas; A second generating module, configured to render the rendering grid based on the downsampling ratio corresponding to the sub-rendering area to generate a first display image; A third generating module, configured to render the rendering grid based on the downsampling ratio and the preset pixel offset corresponding to the sub-rendering area to generate a second display image; The display module is configured to display the first display image through one of the left-eye display and the right-eye display, and display the second display image through the other one of the left-eye display and the right-eye display.

9. A head mounted display, characterized in that: The device comprises a left-eye display, a right-eye display, a memory and a processor; the memory stores a computer program, and the processor implements the display method according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the display method according to any one of claims 1 to 7 is implemented.

Images