Display method, display device, helmet mounted display, and storage medium

Abstract

A display method, a display device (300), a helmet mounted display (100), and a storage medium. The method comprises: rasterizing a scene to be rendered, so as to generate a rendering mesh; on the basis of a target mesh corresponding to a gaze point of human eyes in the rendering mesh, performing partition processing on the rendering mesh to obtain a plurality of sub-rendering areas; rendering the rendering mesh on the basis of the downsampling ratios corresponding to the sub-rendering areas, so as to generate a first display image; rendering the rendering mesh on the basis of the downsampling ratios and preset pixel offsets corresponding to the sub-rendering areas, so as to generate a second display image; and displaying the first display image by means of one of a left-eye display (20) and a right-eye display (30) in the helmet mounted display (100), and displaying the second display image by means of the other.

Description

Display method, display device, head-mounted display and storage medium

[0001] Priority information

[0002] This application claims priority and benefits to patent application No. 202410796783.2, filed with the China National Intellectual Property Administration on June 19, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of virtual reality technology, and in particular relates to a display method, display device, head-mounted display, and non-transitory computer-readable storage medium. Background Technology

[0004] Currently, head-mounted displays (HMDs) provide users with an immersive experience by processing images and projecting the processed images onto the human eye.

[0005] However, due to the size limitations of the headset, its image processing performance has a bottleneck, and the computing power for image rendering has an upper limit. Therefore, for high-resolution display images, it is difficult to achieve a rendering speed that matches the display refresh rate. Conversely, if the rendering speed is to be improved to match the display refresh rate, some resolution must be sacrificed. Summary of the Invention

[0006] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a display method, display device, head-mounted display, and non-transitory computer-readable storage medium that can guarantee the resolution of the displayed image while matching the rendering speed and the display refresh rate.

[0007] The display method of this application is applied to a head-mounted display, which includes a left-eye display and a right-eye display. The method includes: rasterizing a scene to be rendered to generate a rendering mesh; partitioning the rendering mesh into multiple sub-rendering regions based on a target mesh corresponding to the gaze point of the human eye in the rendering mesh, wherein the downsampling factor of each sub-rendering region composed of the target mesh is the minimum downsampling factor of each sub-rendering region; rendering the rendering mesh based on the downsampling factor corresponding to the sub-rendering region to generate a first display image; rendering the rendering mesh based on the downsampling factor corresponding to the sub-rendering region and a preset pixel offset to generate a second display image; and displaying the first display image on one of the left-eye display and the second display image on the other.

[0008] In some implementations, the downsampling rate during rendering of the sub-rendering region is proportional to the distance between the sub-rendering region and the target mesh.

[0009] In some implementations, the preset pixel offset of the sub-rendering region is proportional to the downsampling rate during rendering of the sub-rendering region.

[0010] In some implementations, the preset pixel offset corresponding to the sub-rendering region is equal to half of the downsampling factor of the sub-rendering region.

[0011] In some implementations, the preset pixel offset of the sub-rendering region containing the target mesh is equal to 0.

[0012] In some implementations, rendering the rendering grid based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region to generate a second display image includes: determining an offset grid corresponding to any grid in the sub-rendering region based on the preset pixel offset; and rendering the any grid based on the downsampling ratio and image information corresponding to the offset grid to generate the second display image.

[0013] In some implementations, the offset direction corresponding to the preset pixel offset is at least one of the horizontal direction, the vertical direction, and the diagonal direction.

[0014] The display device of this application is applied to a head-mounted display, which includes a left-eye display and a right-eye display. The display device includes a first generation module, a processing module, a second generation module, a third generation module, and a display module. The first generation module is used to rasterize the scene to be rendered to generate a rendering mesh; the processing module is used to partition the rendering mesh based on the target mesh corresponding to the gaze point of the human eye in the rendering mesh to obtain multiple sub-rendering regions, wherein the downsampling factor of the sub-rendering region composed of the target mesh is the minimum value of the downsampling factor of each sub-rendering region; the second generation module is used to render the rendering mesh based on the downsampling factor corresponding to the sub-rendering region to generate a first display image; the third generation module is used to render the rendering mesh based on the downsampling factor corresponding to the sub-rendering region and a preset pixel offset to generate a second display image; the display module is used to display the first display image through one of the left-eye display and the right-eye display, and display the second display image through the other.

[0015] The head-mounted display of this application includes a left-eye display, a right-eye display, a memory, and a processor; the memory stores a computer program, and the processor executes the program to implement the display method described in any of the above embodiments.

[0016] The non-transitory computer-readable storage medium of this application stores a computer program thereon, which, when executed by a processor, implements the display method as described in any of the above embodiments.

[0017] The display method, display device, head-mounted display, and non-transitory computer-readable storage medium provided in this application render a rendering mesh based on the downsampling ratio corresponding to each sub-rendering region to generate a first display image, and render the rendering mesh based on the downsampling ratio corresponding to the sub-rendering region and a preset pixel offset to generate a second display image, thereby improving the rendering speed of the first and second display images; furthermore, by displaying the first display image on one of the left and right eye displays of the head-mounted display and the second display image on the other, the first and second display images are shifted and superimposed in the human eye, allowing the human eye to perceive more scene details, thereby improving resolution and avoiding color abrupt changes that affect the user experience, that is, ensuring image resolution while improving image rendering speed. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0019] Figure 1 is an application scenario diagram of the display method provided in the embodiment of this application;

[0020] Figure 2 is a flowchart illustrating the display method provided in an embodiment of this application;

[0021] Figure 3 is a schematic diagram of a scenario for the display method provided in an embodiment of this application;

[0022] Figure 4 is a schematic diagram of a scenario for the display method provided in an embodiment of this application;

[0023] Figure 5 is a flowchart illustrating the display method provided in an embodiment of this application;

[0024] Figure 6 is a flowchart illustrating the display method provided in an embodiment of this application;

[0025] Figure 7 is a schematic diagram of the display device provided in an embodiment of this application;

[0026] Figure 8 is a schematic diagram of the connection state between a non-volatile computer-readable storage medium and a processor according to certain embodiments of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0028] To facilitate understanding, the technical background and application scenarios of this application will be introduced below:

[0029] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating a virtual environment that allows for human-computer interaction. XR includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

[0030] Augmented Reality (AR) is a technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing to simulate and apply computer-generated text, images, 3D models, music, videos, and other virtual information to the real world. The two types of information complement each other, thereby "enhancing" the real world.

[0031] Virtual Reality (VR) technology, also known as virtual reality or virtual reality technology, encompasses computer science, electronic information, and simulation technology. Its basic implementation relies primarily on computer technology, utilizing and integrating the latest advancements in 3D graphics, multimedia, simulation, display, and server technologies. The graphics processing unit (GPU) within VR devices processes images of the current scene to create a realistic 3D virtual world offering multiple sensory experiences, including visual, tactile, and olfactory sensations, thus giving viewers a sense of immersion.

[0032] Mixed Reality (MR) technology refers to a new visualization environment created by merging the real and virtual worlds. In this new visualization environment, physical and digital objects coexist and can interact with the real world in real time and obtain information promptly.

[0033] To more vividly illustrate the various technical solutions in the embodiments of this disclosure, the image processing method of this application will be described below using Virtual Reality (VR) and head-mounted displays as examples of VR headsets applied in VR. It is understood that the principle of implementing the image processing method of this application is basically similar for XR technology, and will not be repeated here. It should be emphasized that this is only an example and not a specific limitation on the scope of application of the embodiments of this disclosure.

[0034] Please refer to Figure 1, which is an application scenario diagram of an image processing method provided in an embodiment of this application. The application scenario provided in this application includes a head-mounted display 100 (such as a VR head-mounted display), which includes a body 10, a left-eye display 20, and a right-eye display 30. The left-eye display 20 displays the left-eye image, and the right-eye display 30 displays the right-eye image.

[0035] In one embodiment, the head-mounted display 100 may be a video projection head-mounted display, a holographic head-mounted display, an AR head-mounted display, a VR head-mounted display, etc.

[0036] In one embodiment, the head-mounted display 100 may include at least one image acquisition device 40 (such as a visible light camera, an infrared camera, a structured light camera, etc.) for acquiring two-dimensional or three-dimensional images of the actual scene.

[0037] In one embodiment, the head-mounted display 100 includes at least one of an image signal processing unit (ISP) 50 and a graphics processing unit (GPU) 60. The ISP 50 can be used to process the image signal of the image acquisition device 40, such as performing color correction, noise reduction, gamma correction, etc., to improve the image quality.

[0038] GPU60 can render virtual scenes, such as vertex processing, rasterization, and pixel shading, to obtain virtual scene images.

[0039] Vertex processing involves converting the coordinates of each vertex in the 3D space into the 2D space of the display through linear algebra calculations, and simultaneously calculating the vertex color for subsequent pixel color interpolation operations.

[0040] In the rasterization process, the primitive information generated during vertex processing is input into the rasterization stage. Primitives are first assembled, and then the fragments corresponding to the screen pixels covered by the primitives are determined by triangle traversal. After the rasterization stage, the primitives are divided into basic units of pixel size, which are called fragments. Fragments are more like the data representation of pixels. The final pixels are generated from the information in the fragments. Then, the fragment shader colors the fragments. During the per-fragment operation, the fragments undergo a series of tests. Fragments that pass the tests are converted into pixels and finally presented in the frame buffer area.

[0041] Pixel coloring processing can involve calculating the color information corresponding to each pixel, i.e., the pixel value.

[0042] In one embodiment, the head-mounted display 400 can also fuse the virtual scene image rendered by the GPU 50 with the image processed by the ISP 60 to obtain a fused image, which is then displayed on the left-eye and right-eye displays to achieve an augmented reality visual effect.

[0043] Based on the above description of the relevant scenarios, this application provides a display method, which will be described in detail below:

[0044] Please refer to Figure 2, which is a flowchart illustrating the display method provided in this embodiment. The display method provided in this embodiment is applied to a head-mounted display, which includes a left-eye display and a right-eye display. The display method is implemented by steps 011, 012, 013, 014, and 015. Taking a VR device as an example, the following is a detailed description.

[0045] Step 011: Rasterize the scene to be rendered to generate a rendering mesh.

[0046] The scene to be rendered can be a scene or image that can be displayed on the left or right eye monitor after rendering. The scene to be rendered can be a real scene or a virtual scene, etc.; the rendering mesh is the mesh formed by fragments after the scene to be rendered is rasterized.

[0047] Specifically, a first visual sensor (i.e., an image acquisition device) can be set on a head-mounted display, and the position and orientation of the first visual sensor can simulate the user's eyes. Based on the position and orientation of the first visual sensor, the scene to be rendered in the current scene, when the user is using a VR device, is determined, and the scene to be rendered is rasterized to generate a rendering mesh.

[0048] For example, a virtual scene can be a three-dimensional (3D) scene or a two-dimensional scene. Taking a 3D scene as an example, the scene to be rendered in the current scene is determined based on the first visual sensor that simulates the user's eye. The GPU of the head-mounted display performs a view transformation on the scene to be rendered, converting the three-dimensional coordinates of the 3D scene map into two-dimensional coordinates. Then, vertex processing and rasterization are performed to obtain the rendering mesh of the scene to be rendered.

[0049] As shown in Figure 3, after rasterizing the scene to be rendered, a rendering grid as shown in Figure 3 can be obtained, with one grid corresponding to one pixel.

[0050] Step 012: Based on the target grid corresponding to the gaze point of the human eye in the rendering grid, the rendering grid is partitioned to obtain multiple sub-rendering regions. The downsampling factor of the sub-rendering region composed of the target grid is the minimum downsampling factor of each sub-rendering region.

[0051] The gaze point can be a point or area that the user's eyes are focused on when using a head-mounted display.

[0052] The target mesh can be the mesh corresponding to the human eye's gaze point in the rendered mesh; or the mesh corresponding to the human eye's gaze point in the rendered mesh and the meshes surrounding that mesh. There can be one or more target meshes, such as a 1x1 mesh, a 2x2 mesh, a 3x3 mesh, etc.

[0053] The downsampling ratio can be 1*1, 2*2, 3*3, etc., and can be used to characterize the degree to which the image resolution is reduced when rendering a rendering area. For example, if the rendering mesh resolution is 100*100, after downsampling and rendering at a downsampling ratio of 1*1, a 100*100 image will be obtained; as another example, if the rendering mesh resolution is 100*100, after downsampling and rendering at a downsampling ratio of 2*2, a 50*50 image will be obtained.

[0054] The downsampling rate corresponds to the shading rate of pixels during rendering; for example, the shading rate equals the downsampling rate. The shading rate refers to the number of grid cells in a single shading region. Each grid cell in a single shading region is shaded based on the same color information. When performing pixel shading, if the shading rate is 1*1, then the number of grid cells in a single shading is 1*1, meaning one grid cell is shading each time; if the shading rate is 2*2, then the number of grid cells in a single shading is 2*2, meaning 2*2 grid cells are shading each time.

[0055] The sub-rendering regions obtained after partitioning can be regular shapes, such as rectangular rings or circular rings, or they can be irregular shapes.

[0056] Specifically, this can be achieved by setting up an infrared camera to capture images of a person's eyeball, and then locating the center of the pupil and other feature points of the eye through methods such as detecting infrared reflection. Based on the located center of the pupil and other feature points, the direction of the eye's gaze can be calculated. According to the direction of the eye's gaze, the user's gaze point position can be calculated using methods such as geometric projection. After determining the gaze point position, it is then mapped onto the rendering mesh generated after the scene to be rendered is rasterized, thus determining the target mesh.

[0057] Then, based on the target grid corresponding to the human eye's gaze point in the rendering grid, when partitioning the rendering grid generated for the scene to be rendered, the target grid can be taken as the center and treated as a sub-rendering region. The distance values ​​between each grid in the rendering grid and the target grid are calculated. Then, by setting multiple distance ranges, the grids whose distance values ​​between each grid and the target grid (such as the number of grids that differ between each grid and the target grid in the horizontal or vertical direction) are within the same distance range are treated as a sub-rendering region, thus partitioning the scene into multiple sub-rendering regions.

[0058] For example, referring to Figure 4, with the target grid b in the rendering grid as the center, the target grid is taken as a sub-rendering region 1. Then, by setting a first distance range (such as [0, 2], in units of grids, i.e., the grids within the range of 0 to 2 grids from the target grid b are taken as the sub-rendering region 2 corresponding to the first distance range), and by setting a second distance range (such as [2, 5], in units of grids, i.e., the grids within the range of 2 to 5 grids from the target grid b are taken as the sub-rendering region 3 corresponding to the second distance range).

[0059] It should be noted that the downsampling factor of the sub-rendering region composed of the target mesh is the minimum downsampling factor among all sub-rendering regions. That is, in the rendered image, the portion corresponding to the sub-rendering region composed of the target mesh has the highest resolution. Since the region where the gaze point is located has the highest resolution, this not only saves computational resources for image transmission and calculation but also ensures a good user experience.

[0060] For example, please refer to Figure 4 again. The downsampling factor of sub-rendering region 1, where the target mesh b is located, is 1*1, which is less than the downsampling factor of sub-rendering region 2 (2*2) and less than the downsampling factor of sub-rendering region 3 (4*4). That is, sub-rendering region 1, which contains the target mesh b, has the highest resolution in the generated image after rendering.

[0061] Optionally, the downsampling factor of the sub-rendering region containing the target mesh is 1*1.

[0062] For example, referring to Figure 4, a downsampling ratio of 1*1 is used to sample sub-rendering region 1 containing the target mesh, sub-rendering region 2 uses a downsampling ratio of 2*2, and sub-rendering region 3 uses a downsampling ratio of 4*4. This ensures that sub-rendering region 1 containing the target mesh b has the lowest downsampling ratio during sampling, meaning that the resolution of the user's gaze point area is the highest in the generated image, thereby improving the user's visual experience.

[0063] Optionally, the downsampling rate during sub-rendering is proportional to the distance between the sub-rendering region and the target mesh.

[0064] Specifically, the downsampling ratio during sub-rendering region rendering increases as the distance between the sub-rendering region and the target mesh increases. The further the sub-rendering region is from the target mesh, the higher the downsampling ratio and the lower the resolution during rendering.

[0065] For example, referring to Figure 4 again, the distance between sub-rendering region 3 and the target mesh (sub-rendering region 1) is greater than the distance between sub-rendering region 2 and the target mesh (sub-rendering region 1). During rendering, the downsampling ratio of sub-rendering region 3 (as shown in Figure 4, the downsampling ratio of sub-rendering region 3 is 3*3) is greater than the downsampling ratio of sub-rendering region 2 (as shown in Figure 4, the downsampling ratio of sub-rendering region 2 is 2*2).

[0066] Step 013: Render the rendering mesh based on the downsampling ratio corresponding to the sub-rendering region to generate the first display image.

[0067] Specifically, given the downsampling ratio corresponding to each sub-rendering region, the shading rate of each sub-rendering region can be determined based on the downsampling ratio. The shading rate is the same as the downsampling rate. According to the shading rate of each sub-rendering region, the mesh of each sub-rendering region is shaded to obtain the pixel value of each sub-rendering region. For example, when the downsampling ratio is 2*2, the shading rate is 2*2, that is, 2*2 meshes are shaded each time. By calculating the pixel value of each sub-rendering region in turn, the first display image can be determined.

[0068] Step 014: Render the rendering mesh based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region to generate the second display image.

[0069] The preset pixel offset can be 1, 2, 3, 4, 5, etc., and the unit is grid. For example, if the preset pixel offset is 1, then the pixel is offset by 1 grid.

[0070] Specifically, when rendering each grid of the rendering mesh to obtain the corresponding pixels, a preset pixel offset can also be considered. For example, based on the image data information of the grids that have a preset pixel offset distance from the current grid to be rendered, the current grid to be rendered is sampled and rendered to obtain the pixels corresponding to the current grid to be rendered.

[0071] For example, referring to Figure 3, let's take the example of a preset pixel offset of one grid along the diagonal of the grid (i.e., offset by one grid in both the horizontal and vertical directions). After rasterizing the scene to be rendered, a 10*10 rendering grid is obtained as shown in Figure 3. When sampling and rendering the rendering grid (m, n) in the m-th row and n-th column, the image information used is the image information of the rendering grid (m+1, n+1) in the (m+1)-th row and n+1-th column. For example, when sampling and rendering the rendering grid (1, 1) in the 1st row and 1st column, the image information used is the image information of the rendering grid (2, 2) in the 2nd row and 2nd column. When sampling and rendering the rendering grid (1, 2) in the 1st row and 2nd column, the image information used is the image information of the rendering grid (2, 3) in the 2nd row and 3rd column. In this way, the rendering grid generated after rasterizing the scene to be rendered can be sampled and rendered sequentially with a preset pixel offset.

[0072] When generating the second display image, in addition to considering the downsampling ratio corresponding to each sub-rendering region, it is also necessary to consider the preset pixel offset of each sub-rendering region. Based on the downsampling ratio and preset pixel offset of the sub-rendering region, the corresponding sub-rendering region is rendered to obtain pixel values ​​that are different from those of the sub-rendering regions without preset pixel offset. The pixel values ​​of each offset sub-rendering region can then determine the second display image.

[0073] Optionally, the preset pixel offset of the sub-rendering region is proportional to the downsampling rate during sub-rendering region rendering.

[0074] Specifically, the higher the downsampling ratio of the sub-rendering region, the larger the size of a single pixel after downsampling. For example, when the downsampling ratio is 2*2, the shading rate is also 2*2, meaning that each time a 2*2 grid is shaded, the four pixels corresponding to the 2*2 grid are treated as one large pixel. When pixels are shifted and stacked, to ensure a good stacking effect and avoid poor image quality caused by stacking scenes with significant differences, the pixel offset distance is generally less than one pixel. Therefore, the preset pixel offset can be directly proportional to the downsampling ratio of the sub-rendering region; that is, the higher the downsampling ratio of the sub-rendering region, the larger the corresponding preset pixel offset.

[0075] Optionally, the preset pixel offset corresponding to the sub-rendering region is equal to half of the downsampling factor of the sub-rendering region to achieve a half-pixel offset.

[0076] Specifically, the preset pixel offset corresponding to the sub-rendering region is equal to half of the downsampling ratio of the sub-rendering region, specifically half of the downsampling ratio in the horizontal or vertical direction. For example, taking the pixel offset in the diagonal direction as an example, when the downsampling ratio of the sub-rendering region is 2P*2P, the preset pixel offset of the sub-rendering region is P when generating the second display image. That is, during sampling, the sub-rendering region is offset by P grids in the horizontal direction and P grids in the vertical direction, thereby achieving half-pixel offset in the horizontal and vertical directions. The effect of displacement superposition is better after the half-pixel offset.

[0077] For example, taking horizontal pixel offset as an example, when the downsampling ratio of the sub-rendering area is 2P*2P, the preset pixel offset of the sub-rendering area is P. That is, during sampling, the sub-rendering area is offset by P grids in the horizontal direction, thereby achieving a half-pixel offset in the horizontal direction. The effect of displacement superposition is better after the half-pixel offset.

[0078] Optionally, the preset pixel offset of the sub-rendering region containing the target mesh is equal to 0.

[0079] Specifically, when the human eye views the first and second displayed images, the sub-rendering areas where the gaze point of the first and second displayed images are located need to be aligned in the human eye to ensure that the image content seen by the left and right eyes corresponds or overlaps visually. That is, the preset pixel offset of the sub-rendering area where the target mesh is located needs to be set to 0, and no pixel offset is applied to the area where the gaze point is located. If the sub-rendering areas where the gaze point of the first and second displayed images are not aligned, the head-mounted display may not provide the user with accurate scene perception, and it may also affect the user experience.

[0080] In addition, since the downsampling ratio of the sub-rendering area where the gaze point is located is low and the resolution is already high, the preset pixel offset of the sub-rendering area where the target mesh is located is 0, which also avoids the problem of image quality reduction caused by pixel offset and ensures the display effect of the gaze point.

[0081] Optionally, the offset direction corresponding to the preset pixel offset is at least one of the horizontal, vertical and diagonal directions.

[0082] Specifically, the horizontal direction can be the row direction of the rendering mesh in Figure 3, the vertical direction is the column direction of the rendering mesh in Figure 3, and the diagonal direction can be the diagonal direction of the mesh in Figure 3.

[0083] Taking a preset pixel offset direction of horizontal to the right and a pixel offset of 1 as an example, when sampling and rendering the rendering mesh (m, n) in the m-th row and n-th column, the image information used is the image information of the rendering mesh (m, n+1) in the m-th row and n+1-th column; taking a preset pixel offset direction of vertical downward and a pixel offset of 2 as an example, when sampling and rendering the rendering mesh (m, n) in the m-th row and n-th column, the image information used is the image information of the rendering mesh (m, n) in the m+2-th row and n-th column; taking a preset pixel offset direction of diagonal from the upper left to the lower right and a pixel offset of 1 as an example, when sampling and rendering the rendering mesh (m, n) in the m-th row and n-th column, the image information used is the image information of the rendering mesh (m+1, n+1) in the m+1-th row and n+1-th column.

[0084] Optionally, the preset pixel offset can be combined with different offset magnitudes in the horizontal, vertical and diagonal directions to achieve offsets at various angles.

[0085] Because of the preset pixel offset, for the same scene to be rendered, a first display image and a second display image with different scene details can be generated respectively. Compared with the absence of a preset pixel offset, where the first display image and the second display image are basically the same, the rendered first display image and the second display image can contain different scene details in the same scene respectively.

[0086] Step 015: Display the first image on one of the left-eye and right-eye displays, and display the second image on the other.

[0087] Specifically, the first display image is displayed on one of the left-eye and right-eye displays of the head-mounted display, while the second display image is displayed on the other. For example, the left-eye display of the head-mounted display displays the first display image, and the right-eye display displays the second display image; or, the left-eye display of the head-mounted display displays the second display image, and the right-eye display displays the first display image.

[0088] The sensitivity of the human visual system to resolution decreases sharply from the center of the gaze towards the periphery of the field of view. Existing head-mounted displays utilize this characteristic of the human eye during rendering, using gaze-point rendering technology to render images, thereby saving computing power and bandwidth.

[0089] Gaze-based rendering is a graphics computing technique that uses downsampling at a gradually increasing rate from the area the viewer is looking at during sampling rendering. In other words, it concentrates rendering resources on the area the viewer is looking at, using high resolution to render that area while other areas are rendered at a lower resolution. This reduces computational power and ensures smooth display.

[0090] However, when rendering the current scene using gaze-based rendering technology, the resolution of areas further away from the gaze point decreases more severely. Therefore, when the gaze point changes, areas far from the gaze point may experience color abrupt changes due to the large downsampling rate, which can affect the user experience.

[0091] This application displays a first image on one of the left and right eye displays of a head-mounted display, and a second image on the other. When a user views the scene to be rendered through the head-mounted display, due to the preset pixel offset between the first and second images during rendering, the resulting first and second images contain different scene details of the same scene to be rendered. The first and second images can be shifted and superimposed by the human eye, allowing the human eye to perceive more scene details. This achieves an effect that is essentially the same as improving resolution through image shifting and superimposition, without significantly affecting the display frame rate, thus improving the display effect.

[0092] In other words, this application renders the rendering mesh based on the downsampling ratio corresponding to each sub-rendering region to generate a first display image, and renders the rendering mesh based on the downsampling ratio corresponding to each sub-rendering region and a preset pixel offset to generate a second display image, thereby improving the rendering speed of the first and second display images. Furthermore, by displaying the first display image on one of the left and right eye displays of the head-mounted display and the second display image on the other, the first and second display images are shifted and superimposed in the human eye, allowing the human eye to perceive more scene details, thereby improving resolution and avoiding color abrupt changes that affect the user experience. That is, while improving the image rendering speed, the image resolution is guaranteed.

[0093] It should be noted that the first and second display images can be generated simultaneously or sequentially. Similarly, the first and second display images can be displayed simultaneously or sequentially. For ease of explanation, this will be illustrated by assuming the first display image is displayed on the left-eye monitor and the second display image is displayed on the right-eye monitor.

[0094] (1) The first display image and the second display image are generated simultaneously, and the first display image is displayed on the left eye display and the second display image is displayed on the right eye display at the same time. That is, steps 013 and 014 are executed simultaneously, and step 015 displays the first display image and the second display image at the same time.

[0095] For example, at time T1, the first and second display images are generated simultaneously, and at time T2, the first display image is displayed on the left eye display and the second display image is displayed on the right eye display.

[0096] (2) The first display image and the second display image are generated simultaneously, and there is a time difference between the first display image and the second display image when they are displayed. The time difference during display is less than the integration time. That is, steps 013 and 014 are executed simultaneously, and step 015 is executed in a time-division manner to display the first display image and the second display image.

[0097] The human eye exhibits the persistence of vision, meaning that an image takes a certain amount of time to solidify on the retina, and the images displayed during this time are superimposed on the retina. Based on this persistence of vision, the human eye perceives the superimposed image within this timeframe.

[0098] Therefore, when the first and second display images are generated simultaneously, and the time difference between their display is less than the integration time, the first and second display images can be superimposed on the human eye, allowing the human eye to perceive more scene details, thereby achieving the effect of improving resolution.

[0099] For example, taking the simultaneous generation of the first and second display images at time T3 as an example, at time T4, the first display image is displayed on the left eye display, and at time T4+ΔT1, the second display image is displayed on the right eye display, where ΔT1 is less than the integration time of the human eye.

[0100] (3) The first display image and the second display image are generated in sequence and displayed simultaneously. The time difference between the generation of the first display image and the second display image is less than the integration time. That is, steps 013 and 014 are executed in time-sharing. Step 015 displays the first display image and the second display image simultaneously.

[0101] Similarly, taking the example of the first display image being generated at time T5 and the second display image being generated at time T5+ΔT2, at time T6, the first display image is displayed on the left-eye monitor, and the second display image is displayed on the right-eye monitor, where ΔT2 is less than the integration time of the human eye. Based on the persistence of vision characteristic of the human eye, the first and second display images can be superimposed in the human eye, thereby improving resolution.

[0102] (4) The first display image and the second display image are generated in sequence and displayed in sequence. The sum of the time difference during generation and the time difference during display is less than the integration time of the human eye. That is, steps 013 and 014 are executed in time-sharing, and steps 015, which display the first display image and display the second display image, are executed in time-sharing.

[0103] Similarly, taking the example of the first displayed image being generated at time T7 and the second displayed image being generated at time T7+ΔT3, the first displayed image is shown on the left eye display at time T8, and the second displayed image is shown on the right eye display at time T8+ΔT4, where ΔT3+ΔT4 is less than the integration time of the human eye. Based on the persistence of vision characteristic of the human eye, the first and second displayed images can be superimposed in the human eye, thereby improving resolution.

[0104] Referring to Figure 5, in some embodiments, step 014: rendering the rendering mesh based on the downsampling ratio corresponding to the sub-rendering region and the preset pixel offset to generate a second display image, includes:

[0105] Step 0141: Based on the preset pixel offset, determine the offset grid corresponding to any grid in the sub-rendering area;

[0106] Step 0142: Based on the downsampling ratio corresponding to the sub-rendering region and the image information corresponding to the offset grid, render any grid to generate a second display image.

[0107] Specifically, please refer to Figure 4. For example, the downsampling ratio of sub-rendering region 1 is 1*1, the downsampling ratio of sub-rendering region 2 is 2*2, and the downsampling ratio of sub-rendering region 3 is 4*4. That is, the shading rate of sub-rendering region 1 is 1*1, the shading rate of sub-rendering region 2 is 2*2, and the shading rate of sub-rendering region 3 is 4*4.

[0108] When generating the first display image, without considering the preset pixel offset, for sub-rendering region 1, 1*1 grids are colored each time; for sub-rendering region 2, 2*2 grids are colored each time; and for sub-rendering region 3, 3*3 grids are colored each time. In this way, the pixel values ​​corresponding to each grid in each sub-rendering region can be obtained, thereby determining the first display image.

[0109] When generating the second display image, the preset pixel offset corresponding to each sub-rendering region needs to be considered. If the currently rendered grid is located in sub-rendering region 1 within the rendering grid, such as target grid b, no pixel offset is performed, and sampling rendering is performed based on the image information of target grid b. If the grid being rendered (such as the grid with coordinates (4, 5) in the rendering grid, i.e., the grid in the 4th row and 5th column) is located in sub-rendering region 2 within the rendering grid, based on the preset pixel offset 1*1, the offset grid corresponding to the grid with coordinates (4, 5) is determined to be the grid with coordinates (5, 6). Based on the downsampling ratio of sub-rendering region 2 2*2 and the image information corresponding to the offset grid with coordinates (5, 6), The grid at coordinates (4, 5) is pixel-colored. If the grid to be rendered (such as the grid at coordinates (1, 1) in the rendering grid, i.e., the grid in the first row and first column) is located in sub-rendering region 3 in the rendering grid, the offset grid corresponding to the grid at coordinates (1, 1) is determined to be the grid at coordinates (3, 3) based on the preset pixel offset 2*2. Based on the downsampling ratio 4*4 and the image information of the offset grid at coordinates (3, 3), the grid at coordinates (1, 1) is pixel-colored. The offset grids corresponding to each grid in each sub-rendering region are determined in turn. Then, based on the downsampling ratio corresponding to each sub-rendering region and the image information corresponding to the offset grid, each grid is rendered to generate the second display image.

[0110] When generating the second display image, the current grid can also be rendered based on the combined image information of the current grid and the grid after a preset pixel offset. If the grid being rendered is located in sub-rendering region 1 within the rendering grid, such as target grid b, no pixel offset is performed; instead, the image information of target grid b is used for sampling and rendering. If the grid being rendered (e.g., the grid with coordinates (4, 5) in the rendering grid, i.e., the grid in the 4th row and 5th column) is located in sub-rendering region 2 within the rendering grid, then based on the downsampling ratio of 2*2 and the preset pixel offset of 1*1, the image information of the grid with coordinates (5, 6) and the grid with coordinates (4, 5) can be combined to render the grid with coordinates (4, 6). 5) The grid is pixel-colored; if the grid to be rendered (such as the grid with coordinates (1,1) in the rendering grid, i.e. the grid in the first row and first column) is located in sub-rendering region 3 in the rendering grid, then based on the downsampling ratio of 4*4 and the preset pixel offset of 2*2, according to the image information of the grid with coordinates (3,3) and the grid with coordinates (1,1) combined, the grid with coordinates (1,1) is pixel-colored, and each grid in each sub-rendering region is rendered with the corresponding downsampling ratio and the corresponding preset pixel offset in turn, thereby generating the second display image.

[0111] When viewing the first and second display images using a head-mounted device, since each sub-rendering area in the second display image, except for the sub-rendering area where the gaze point is located, is sampled with a corresponding preset pixel offset during sampling, the first and second display images respectively contain different scene details of the scene to be rendered. The first and second display images will be shifted and superimposed in the human eye, allowing the human eye to perceive more scene details, achieving an effect that is basically the same as the resolution improvement achieved by image shifting and superimposing, and basically does not affect the display frame rate, thus improving the display effect.

[0112] To facilitate understanding of the display method, the following describes the various steps involved in image generation and display. Please refer to Figure 6, which is a complete flowchart illustrating the generation and display of the first and second display images in the display method provided in this embodiment. Taking the first display image displayed on the left-eye monitor and the second display image displayed on the right-eye monitor as an example, the display method includes:

[0113] Step 021: Divide the rendering mesh into multiple sub-rendering regions based on the gaze point location;

[0114] Step 022: Based on the downsampling ratio corresponding to the sub-rendering region, render the corresponding sub-rendering region to generate the first display image;

[0115] Step 023: Display the first image on the left-eye monitor;

[0116] Step 024: Based on the downsampling ratio and preset pixel offset of the sub-rendering region, render the corresponding sub-rendering region to generate the second display image;

[0117] Step 025: Display the second image on the right eye monitor.

[0118] For a detailed description of step 021, please refer to the descriptions of steps 011 and 012; for a detailed description of step 022, please refer to the description of step 013; for a detailed description of step 023, please refer to the description of step 015; for a detailed description of step 024, please refer to the descriptions of steps 014, 0141, and 0142; and for a detailed description of step 025, please refer to the description of step 015. For the sake of brevity, these details will not be repeated here.

[0119] According to the method described in the above embodiments, this application also provides a display device 300 for performing the steps in the above display method. Please refer to FIG7, which is a schematic structural diagram of the display device 300 provided in this application embodiment. The display device 300 includes a first generation module 301, a processing module 302, a second generation module 303, a third generation module 304, and a display module 305, wherein:

[0120] The first generation module 301 is used to rasterize the scene to be rendered in order to generate a rendering mesh;

[0121] The processing module 302 is used to partition the rendering mesh based on the target mesh corresponding to the gaze point of the human eye in the rendering mesh to obtain multiple sub-rendering regions. The downsampling factor of the sub-rendering region containing the target mesh is the minimum value of the downsampling factor of each sub-rendering region.

[0122] The second generation module 303 is used to render the rendering mesh based on the downsampling ratio corresponding to the sub-rendering region to generate the first display image;

[0123] The third generation module 304 is used to render the rendering mesh based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region to generate a second display image;

[0124] The display module 305 is used to display a first display image on one of the left-eye display and the right-eye display, and to display a second display image on the other.

[0125] It should be noted that the specific details of each module unit in the above-mentioned display device 300 have been described in detail in the embodiments of the above-mentioned display method, and will not be repeated here.

[0126] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0127] In some embodiments, the display device 300 in this application can be implemented in hardware, such as an electronic device or a component in an electronic device, such as an integrated circuit or a chip; the display device 300 can also be implemented in software, such as as an application installed in an electronic device.

[0128] In some embodiments, please refer to FIG1, which is a schematic diagram of the structure of a head-mounted display provided in an embodiment of this application. The head-mounted display 100 includes a processor 70, a memory 80, a left-eye display 20, and a right-eye display 30. The memory 80 stores a computer program 81 that can run on the processor 70. When the processor 70 executes the program 81, it implements the various processes of the above-described display method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0129] This application also provides a non-transitory computer-readable storage medium. Please refer to FIG8. The non-transitory computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described display method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0130] The processor can be the processor in the electronic device described in the above embodiments. The computer-readable storage medium can be a computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0131] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A display method, characterized in that, Applied to a head-mounted display, the head-mounted display including a left-eye display and a right-eye display, the method includes: The scene to be rendered is rasterized to generate a rendering mesh; Based on the target grid corresponding to the gaze point of the human eye in the rendering grid, the rendering grid is partitioned to obtain multiple sub-rendering regions. The downsampling factor of the sub-rendering region composed of the target grid is the minimum downsampling factor of each sub-rendering region. Based on the downsampling ratio corresponding to the sub-rendering region, the rendering mesh is rendered to generate a first display image; Based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region, the rendering mesh is rendered to generate a second display image; The first display image is displayed on one of the left-eye and right-eye displays, and the second display image is displayed on the other.

2. The display method according to claim 1, characterized in that, The downsampling rate during rendering of the sub-rendering region is proportional to the distance between the sub-rendering region and the target mesh.

3. The display method according to claim 1 or 2, characterized in that, The preset pixel offset of the sub-rendering region is proportional to the downsampling ratio during rendering of the sub-rendering region.

4. The display method according to claim 1 or 3, characterized in that, The preset pixel offset corresponding to the sub-rendering region is equal to 1 / 2 of the downsampling ratio of the sub-rendering region.

5. The display method according to claim 3 or 4, characterized in that, The preset pixel offset of the sub-rendering region containing the target mesh is equal to 0.

6. The display method according to claim 1, characterized in that, The step of rendering the rendering mesh based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region to generate a second display image includes: Based on the preset pixel offset, determine the offset grid corresponding to any grid in the sub-rendering area; Based on the downsampling ratio corresponding to the sub-rendering region and the image information corresponding to the offset grid, render any one of the grids to generate the second display image.

7. The display method according to any one of claims 1-6, characterized in that, The preset pixel offset corresponds to an offset direction that is at least one of the horizontal, vertical, and diagonal directions.

8. A display device, characterized in that, For use in head-mounted displays, the head-mounted display including a left-eye display and a right-eye display, the device includes: The first generation module is used to rasterize the scene to be rendered in order to generate a rendering mesh; The processing module is used to partition the rendering grid based on the target grid corresponding to the gaze point of the human eye in the rendering grid to obtain multiple sub-rendering regions. The downsampling factor of the sub-rendering region composed of the target grid is the minimum value of the downsampling factor of each sub-rendering region. The second generation module is used to render the rendering mesh based on the downsampling ratio corresponding to the sub-rendering region to generate a first display image; The third generation module is used to render the rendering grid based on the downsampling ratio and preset pixel offset corresponding to the sub-rendering region to generate a second display image; The display module is used to display the first display image through one of the left-eye display and the right-eye display, and to display the second display image through the other.

9. A head-mounted display, characterized in that, It includes a left-eye display, a right-eye display, a memory, and a processor; the memory stores a computer program, and the processor executes the program to implement the display method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the display method as described in any one of claims 1-7.

Images