Image display methods, devices and computer-readable storage media

Abstract

This invention discloses an image display method, device, and computer-readable storage medium. The method includes: determining a gaze region and a non-gaze region in a screen to be displayed, wherein the non-gaze region is the area in the screen to be displayed other than the gaze region; determining an initial display resolution corresponding to the non-gaze region based on a preset resolution of the display screen and a display resolution corresponding to the gaze region; determining a pixel magnification ratio corresponding to the non-gaze region based on the actual resolution corresponding to the non-gaze region and the initial display resolution, wherein the actual resolution corresponding to the gaze region is greater than the actual resolution corresponding to the non-gaze region; adjusting the current pixel size of the non-gaze region to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is greater than the current pixel size; and controlling a head-mounted display device to display the screen to be displayed after adjusting the pixel size, thereby improving the display effect of the head-mounted display device.

Description

Technical Field

[0001] This invention relates to the field of head-mounted display technology, and more particularly to an image display method, device, and computer-readable storage medium. Background Technology

[0002] With the development of image processing technology, people have increasingly higher requirements for the display effects of head-mounted display devices. To meet user needs, manufacturers are pursuing even more demanding display effects. Related technologies involve rendering captured images and then transmitting the rendered images to the display screen. However, when the image resolution is high, the amount of image data transmitted increases, and the transmission rate slows down, leading to a deterioration in the display effect of the head-mounted display device. Summary of the Invention

[0003] This application provides an image display method, device, and computer-readable storage medium, aiming to improve the display effect of head-mounted display devices.

[0004] This application provides an image display method applied to a head-mounted display device, the image display method comprising:

[0005] Determine the gaze area and non-gaze area in the image to be displayed, wherein the non-gaze area is the area in the image to be displayed other than the gaze area;

[0006] The initial display resolution corresponding to the non-focused area is determined based on the preset resolution of the display screen and the display resolution corresponding to the gaze area.

[0007] The pixel magnification ratio of the non-focused area is determined based on the actual resolution corresponding to the non-focused area and the initial display resolution, wherein the actual resolution corresponding to the focused area is greater than the actual resolution corresponding to the non-focused area.

[0008] Based on the pixel magnification ratio, the current pixel size of the non-focused region is adjusted to the target pixel size, where the target pixel size is larger than the current pixel size;

[0009] Control the head-mounted display device to display the image to be displayed after adjusting the pixel size.

[0010] Optionally, before the step of determining the initial display resolution corresponding to the non-focused area based on the preset resolution of the display screen and the display resolution corresponding to the gaze area, the method further includes:

[0011] A first transmission rate for the gaze region and a second transmission rate for the non-gaze region are determined, wherein the first transmission rate is greater than the second transmission rate;

[0012] Determine the current screen refresh rate and color depth;

[0013] Based on the first transmission rate, the current screen refresh rate, and the color depth, the display resolution corresponding to the gaze area is determined, and based on the second transmission rate, the current screen refresh rate, and the color depth, the actual resolution corresponding to the non-gaze area is determined.

[0014] Optionally, the step of determining the display resolution corresponding to the gaze area based on the first transmission rate, the current screen refresh rate, and the color depth, and determining the actual resolution corresponding to the non-gaze area based on the second transmission rate, the current screen refresh rate, and the color depth, includes:

[0015] A first resolution is determined based on the ratio of the first transmission rate to the current screen refresh rate. A second resolution is determined based on the ratio of the first resolution to the color depth. The second resolution is then square-rooted to obtain the display resolution corresponding to the gaze area.

[0016] Based on the ratio of the second transmission rate to the current screen refresh rate, a third resolution is determined. Based on the ratio of the third resolution to the color depth, a fourth resolution is determined. The square root of the fourth resolution is then performed to obtain the actual resolution corresponding to the non-focused area.

[0017] Optionally, the step of determining the first transmission rate of the gaze region and the second transmission rate of the non-gaze region includes:

[0018] Get the current system's maximum transmission rate;

[0019] Determine the transmission rate ratio corresponding to the gaze region and the non-gaze region;

[0020] Based on the maximum transmission rate and the transmission rate percentage, a first transmission rate corresponding to the gaze region and a second transmission rate corresponding to the non-gaze region are determined.

[0021] Optionally, the step of determining the initial display resolution corresponding to the non-focused area based on the preset resolution of the display screen and the display resolution corresponding to the gaze area includes:

[0022] The difference between the preset resolution of the display screen and the display resolution corresponding to the gaze area is determined as the initial display resolution corresponding to the non-gaze area.

[0023] Optionally, the step of determining the pixel magnification ratio corresponding to the non-focused region based on the actual resolution corresponding to the non-focused region and the initial display resolution includes:

[0024] The ratio of the actual resolution corresponding to the non-focused area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-focused area.

[0025] Optionally, determining the gaze area in the image to be displayed includes:

[0026] Determine the pupil position at the moment the image to be displayed will be shown;

[0027] The gaze area in the image to be displayed is determined based on the pupil position.

[0028] Optionally, the step of determining the pupil position at the moment the image to be displayed is shown includes:

[0029] Determine the difference in relative position between the historical pupil position and the initial pupil position;

[0030] The relative offset and direction of the pupil are determined based on the difference in relative position.

[0031] The pupil position is determined based on the relative offset and the offset direction.

[0032] In addition, to achieve the above objectives, the present invention also provides a head-mounted display device, the head-mounted display device comprising: a memory, a processor, and an image display program stored in the memory and executable on the processor, wherein the image display program, when executed by the processor, implements the steps of the image display method described above.

[0033] In addition, to achieve the above objectives, the present invention also provides a storage medium storing an image display program thereon, which, when executed by a processor, implements the steps of the image display method described above.

[0034] This application provides a technical solution for an image display method, device, and computer-readable storage medium. The method divides the image to be displayed into a gaze area that requires high-definition display and a non-gaze area that does not require high-definition display. Based on the preset resolution of the display screen and the display resolution corresponding to the gaze area, the initial display resolution corresponding to the non-gaze area is determined. The gaze area is displayed using the actual resolution. The current pixel size of the non-gaze area is adjusted to a target pixel size that is larger than the pixel size. The non-gaze area is blurred, reducing the data transmission volume of the non-gaze area and improving the data transmission efficiency of the non-gaze area, thereby improving the display effect of the head-mounted display device. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of the head-mounted display device according to an embodiment of the present invention;

[0036] Figure 2This is a flowchart illustrating the first embodiment of the image display method of the present invention;

[0037] Figure 3 This is a schematic diagram of the first application scenario involved in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the second application scenario involved in an embodiment of the present invention;

[0039] Figure 5 Diagram showing the fixational and non-fixational regions;

[0040] Figure 6 This is a schematic diagram of an embodiment of the present invention.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings are only one embodiment and not the entirety of the invention. Detailed Implementation

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] In related technologies, the captured image is rendered and then transmitted to a display screen. When the image resolution is high, the amount of image data transmitted increases, and the transmission rate is slow, resulting in a deterioration in the display effect of the head-mounted display device.

[0044] To improve the display effect of head-mounted display devices, this invention proposes a control method for head-mounted display devices. The main steps of the method include: determining a gaze region and a non-gaze region in a display image, wherein the non-gaze region is the area in the display image other than the gaze region; determining an initial display resolution for the non-gaze region based on a preset resolution of the display screen and the display resolution corresponding to the gaze region; determining a pixel magnification ratio for the non-gaze region based on the actual resolution corresponding to the non-gaze region and the initial display resolution, wherein the actual resolution corresponding to the gaze region is greater than the actual resolution corresponding to the non-gaze region; adjusting the current pixel size of the non-gaze region to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is greater than the current pixel size; and controlling the head-mounted display device to display the display image after the pixel size adjustment. This method divides the screen to be displayed into a gaze area that requires high-definition display and a non-gaze area that does not require high-definition display. Based on the preset resolution of the display screen and the display resolution corresponding to the gaze area, the initial display resolution corresponding to the non-gaze area is determined. The gaze area is displayed using the actual resolution, while the current pixel size of the non-gaze area is adjusted to a target pixel size that is larger than the pixel size. The non-gaze area is then blurred, reducing the amount of data transmission in the non-gaze area and improving the data transmission efficiency of the non-gaze area, thereby improving the display effect of the head-mounted display device.

[0045] Furthermore, this application determines the display resolution of the content corresponding to the gaze area and the actual resolution corresponding to the non-gaze area based on a first transmission rate of the gaze area, a second transmission rate of the non-gaze area, the current screen refresh rate, and color depth. This allows the gaze area and the non-gaze area to be divided according to the transmission rate. When the screen refresh rate and resolution are low (i.e., when MIPI has sufficient transmission capacity), the display area of ​​the gaze area is expanded, and the displayed content is clearer. When the screen refresh rate and resolution are high (i.e., when MIPI lacks sufficient transmission capacity), the display area of ​​the gaze area is reduced to meet transmission capacity requirements. While ensuring transmission capacity, the pixels in the gaze area are more densely packed, allowing for clear display of the content in the gaze area.

[0046] The claims of this invention will be described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0048] In this embodiment of the invention, the terminal can be a head-mounted display device.

[0049] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The memory 1003 may be a high-speed RAM or a stable memory, such as a disk drive. Optionally, the memory 1003 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] like Figure 1 As shown, the memory 1003, which serves as a computer storage medium, may include an operating system and a control program for the head-mounted display device.

[0052] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0053] Determine the gaze area and non-gaze area in the image to be displayed, wherein the non-gaze area is the area in the image to be displayed other than the gaze area;

[0054] The initial display resolution corresponding to the non-focused area is determined based on the preset resolution of the display screen and the display resolution corresponding to the gaze area.

[0055] The pixel magnification ratio of the non-focused area is determined based on the actual resolution corresponding to the non-focused area and the initial display resolution, wherein the actual resolution corresponding to the focused area is greater than the actual resolution corresponding to the non-focused area.

[0056] Based on the pixel magnification ratio, the current pixel size of the non-focused region is adjusted to the target pixel size, where the target pixel size is larger than the current pixel size;

[0057] Control the head-mounted display device to display the image to be displayed after adjusting the pixel size.

[0058] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0059] A first transmission rate for the gaze region and a second transmission rate for the non-gaze region are determined, wherein the first transmission rate is greater than the second transmission rate;

[0060] Determine the current screen refresh rate and color depth;

[0061] Based on the first transmission rate, the current screen refresh rate, and the color depth, the display resolution corresponding to the gaze area is determined, and based on the second transmission rate, the current screen refresh rate, and the color depth, the actual resolution corresponding to the non-gaze area is determined.

[0062] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0063] A first resolution is determined based on the ratio of the first transmission rate to the current screen refresh rate. A second resolution is determined based on the ratio of the first resolution to the color depth. The second resolution is then square-rooted to obtain the display resolution corresponding to the gaze area.

[0064] Based on the ratio of the second transmission rate to the current screen refresh rate, a third resolution is determined. Based on the ratio of the third resolution to the color depth, a fourth resolution is determined. The square root of the fourth resolution is then performed to obtain the actual resolution corresponding to the non-focused area.

[0065] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0066] Get the current system's maximum transmission rate;

[0067] Determine the transmission rate ratio corresponding to the gaze region and the non-gaze region;

[0068] Based on the maximum transmission rate and the transmission rate percentage, a first transmission rate corresponding to the gaze region and a second transmission rate corresponding to the non-gaze region are determined.

[0069] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0070] The difference between the preset resolution of the display screen and the display resolution corresponding to the gaze area is determined as the initial display resolution corresponding to the non-gaze area.

[0071] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0072] The ratio of the actual resolution corresponding to the non-focused area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-focused area.

[0073] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0074] Determine the pupil position at the moment the image to be displayed will be shown;

[0075] The gaze area in the image to be displayed is determined based on the pupil position.

[0076] The processor 1001 can be used to call the control program of the head-mounted display device stored in the memory 1003 and perform the following operations:

[0077] Determine the difference in relative position between the historical pupil position and the initial pupil position;

[0078] The relative offset and direction of the pupil are determined based on the difference in relative position.

[0079] The pupil position is determined based on the relative offset and the offset direction.

[0080] like Figure 2 As shown, in the first embodiment of this application, the image display method of this application includes the following steps:

[0081] Step S110: Determine the gaze area and non-gaze area in the screen to be displayed, wherein the non-gaze area is the area in the screen to be displayed other than the gaze area.

[0082] In this embodiment, a head-mounted display device refers to a display device worn on a user's head, such as a virtual display device or an augmented reality display device. When a user views the content displayed on the screen of a head-mounted display device at close range, the field of view is relatively small and may not completely encompass the screen. Therefore, partial high-definition display can reduce the resolution of display content that the user cannot see or cannot see clearly, while preserving the display effect of the display content that the user can see or is of primary interest to. In this way, without changing the user's viewing experience, the product's power consumption can be reduced.

[0083] In this embodiment, the gaze area is the position where the eye looks at the screen at the moment the image to be displayed is shown. This includes the center of the gaze area and some surrounding pixels. The number of surrounding pixels depends on the screen refresh rate and resolution, which is related to the MIPI transmission capability. When the screen refresh rate and resolution are low, meaning the MIPI has sufficient transmission capability, the display range of the surrounding pixels is expanded. When the screen refresh rate and resolution are high, meaning the MIPI does not have sufficient transmission capability, the display range of the surrounding pixels is reduced to meet transmission requirements. The non-gaze area refers to the area in the image to be displayed other than the gaze area, i.e., the position where the eye is not directly looking at the screen, specifically the pixel area other than the center and surrounding pixels of the gaze area.

[0084] For reference Figure 5 Based on the human eye's field of view and pupil position, the area that the user can see is determined as a square region Q, and the area in the image to be displayed other than the square region Q is determined as the non-focused area.

[0085] Step S120: Determine the initial display resolution corresponding to the non-focused area based on the preset resolution of the display screen and the display resolution corresponding to the gaze area.

[0086] In this embodiment, different display models have different preset resolutions. These preset resolutions can be fixed or multiple can be pre-set, with the appropriate resolution selected based on different application scenarios. For example, the current display's preset resolution can be 4000*4000. The display resolution corresponding to the viewing area refers to the display resolution of the content displayed in the viewing area. The display resolution corresponding to the viewing area can be calculated based on the first transmission rate of the viewing area, the current screen refresh rate, and the color depth. This ensures that when the screen refresh rate and resolution are low (i.e., when MIPI has sufficient transmission capability), the display range of the viewing area expands, and the displayed content is clearer. When the screen refresh rate and resolution are high (i.e., when MIPI lacks sufficient transmission capability), the display range of the viewing area shrinks to meet transmission requirements. While ensuring transmission capability, the pixels in the viewing area are denser, allowing for clear display of the content in the viewing area.

[0087] In this embodiment, since the display content in the gaze area consumes more resources and renders as much image data as possible, this application sets the correspondence between the displayed pixels and actual pixels of the display content corresponding to the gaze area to 1:1. This means the displayed pixels and actual pixels of the display content are set to be the same, i.e., the actual resolution of the gaze area is used as the display resolution of the display content corresponding to the gaze area, restoring the display content corresponding to the gaze area one-to-one without compression, so that the display content in the user's gaze area can be clearly displayed. Optionally, the difference between the preset resolution of the display screen and the display resolution corresponding to the gaze area can be determined as the initial display resolution corresponding to the non-gaze area, thereby determining the number of pixels required to be displayed in the non-gaze area. For example, if the current preset resolution of the display screen is 4000*4000 and the display resolution corresponding to the gaze area is 1848, then the initial display resolution corresponding to the non-gaze area is 4000-1848=2152.

[0088] Step S130: Determine the pixel magnification ratio of the non-focused area based on the actual resolution corresponding to the non-focused area and the initial display resolution, wherein the actual resolution corresponding to the focused area is greater than the actual resolution corresponding to the non-focused area.

[0089] In this embodiment, the actual resolution corresponding to the non-focused area can be calculated based on the second transmission rate of the non-focused area, the current screen refresh rate, and the color depth. This application sets the correspondence between the displayed pixels and actual pixels of the content displayed in the non-focused area to N:1, where N is the pixel magnification ratio, and N is greater than or equal to 2.

[0090] Optionally, the ratio of the actual resolution corresponding to the non-focused area to the initial display resolution is determined as the pixel magnification ratio for the non-focused area. For example, assuming the actual resolution of the non-focused area is calculated to be 924 based on the second transmission rate of the non-focused area, the current screen refresh rate, and the color depth, and the initial display resolution of the non-focused area is 2152, then the pixel magnification ratio for the non-focused area is N = 2152 / 924 = 2.3. Rounding down, N = 3, meaning that three adjacent pixels share the same display data. Since displaying content in the non-focused area consumes fewer resources and reduces data rendering, setting the ratio of pixels in areas not focused on by the eye to actual pixels to N:1 displays the content of the center pixel within a range of N pixels surrounding a given pixel. This reduces the amount of data rendered, thereby reducing data transmission and improving the image display effect.

[0091] Step S140: Based on the pixel magnification ratio, adjust the current pixel size of the non-focused region to the target pixel size, wherein the target pixel size is larger than the current pixel size.

[0092] In this embodiment, the non-focused area of ​​the image to be displayed is determined based on the gaze area. The image to be displayed is composed of pixels arranged in a grid. The pixel size of the pixels in the image to be displayed is the current pixel size. The current pixel size is adjusted to the target pixel size, which is larger than the current pixel size. After adjusting to the target pixel size, the number of pixels per unit area in the non-focused area becomes smaller, which reduces the clarity in the non-focused area and reduces the power consumption during display.

[0093] Optionally, the gaze area occupied by the target pixel in the non-gaze area after adjusting to the target pixel size is determined, and the current pixels in the non-gaze area within the gaze area are merged into the target pixel. During merging, the target pixel value of the target pixel is determined according to the pixel value corresponding to the current pixel in the non-gaze area within the gaze area. For example, the average value of the pixel value corresponding to the current pixel in the non-gaze area within the gaze area is used as the target pixel value. The target image corresponding to the non-gaze area is constructed based on the target pixel value and the gaze area.

[0094] Step S150: Control the head-mounted display device to display the image to be displayed after adjusting the pixel size.

[0095] In this embodiment, after adjusting the pixel size of the non-focused area to the target pixel size, a target image of the non-focused area is generated. The target image obtained after adjusting the pixel size and the image corresponding to the focused area without adjusting the pixel size are combined into an image to be displayed after adjusting the pixel size.

[0096] Optionally, the images of the non-focused region and the focused region of the target pixels are rendered separately, and the rendered images are then stitched together to obtain the stitched image to be displayed. The stitched image is then transmitted to the screen via MIPI for display.

[0097] Because the current pixel size of the non-focused area is adjusted to a larger target pixel size, the number of pixels per unit area in the non-focused area decreases, resulting in reduced clarity in the non-focused area. However, the corresponding display power consumption is reduced. Meanwhile, the pixel size of the focused area that the user can see or clearly at the time of display remains the current pixel size, and the clarity does not change. In fact, the clarity is higher than that of the non-focused area, so the user's viewing experience is not reduced, thus achieving local high-definition display.

[0098] According to the above technical solution, this embodiment divides the screen to be displayed into a gaze area that requires high-definition display and a non-gaze area that does not require high-definition display. Based on the preset resolution of the display screen and the display resolution corresponding to the gaze area, the initial display resolution corresponding to the non-gaze area is determined. The gaze area is displayed using the actual resolution. The current pixel size of the non-gaze area is adjusted to a target pixel size that is larger than the pixel size. The non-gaze area is blurred to reduce the data transmission volume of the non-gaze area and improve the data transmission efficiency of the non-gaze area, thereby improving the display effect of the head-mounted display device.

[0099] Further, prior to step S120, the following steps are included:

[0100] Step S210: Determine a first transmission rate for the gaze region and a second transmission rate for the non-gaze region, wherein the first transmission rate is greater than the second transmission rate.

[0101] In this embodiment, to ensure stable transmission, the first transmission rate and the second transmission rate are different. Since the required clarity for the gaze area is higher than that for the non-gaze area, to improve the image display effect, the first transmission rate corresponding to the gaze area should be set to be greater than the second transmission rate corresponding to the non-gaze area. The first and second transmission rates can be determined based on the current system's maximum transmission rate and the ratio of the transmission rates corresponding to the gaze area and the non-gaze area. This ensures that when the screen refresh rate and resolution are low, i.e., when MIPI has sufficient transmission capacity, the display area of ​​the gaze area is expanded, and the displayed content is clearer. When the screen refresh rate and resolution are high, i.e., when MIPI does not have sufficient transmission capacity, the display area of ​​the gaze area is reduced to meet the transmission capacity requirements. While ensuring transmission capacity, the pixels in the gaze area are denser, enabling clear display of the content in the gaze area. Optionally, the first and second transmission rates can also be preset according to actual conditions.

[0102] Optionally, the maximum transmission rate of the current system is obtained; the transmission rate ratios corresponding to the observed region and the non-observed region are determined; based on the maximum transmission rate and the transmission rate ratios, a first transmission rate corresponding to the observed region and a second transmission rate corresponding to the non-observed region are determined. Assuming the maximum transmission rate of a single MIPI LANE is 1.5G, and to ensure stable transmission, the actual usable maximum transmission rate is 1.1G, then 80% of the bandwidth, i.e., 880M, is allocated to data transmission in the observed region, and the remaining 20%, i.e., 220M, is allocated to data transmission in the non-observed region.

[0103] Step S220: Determine the current screen refresh rate and color depth.

[0104] In this embodiment, the current screen refresh rate and color depth can be preset or fixed as needed, for example, the current screen refresh rate can be set to 90Hz and the color depth can be set to 3.

[0105] Step S230: Determine the display resolution corresponding to the gaze area based on the first transmission rate, the current screen refresh rate, and the color depth; and determine the actual resolution corresponding to the non-gaze area based on the second transmission rate, the current screen refresh rate, and the color depth.

[0106] Optionally, a first resolution is determined based on the ratio of the first transmission rate to the current screen refresh rate; a second resolution is determined based on the ratio of the first resolution to the color depth; and the second resolution is square-rooted to obtain the display resolution corresponding to the gaze area; and a third resolution is determined based on the ratio of the second transmission rate to the current screen refresh rate; a fourth resolution is determined based on the ratio of the third resolution to the color depth; and the fourth resolution is square-rooted to obtain the actual resolution corresponding to the non-gaze area.

[0107] Assuming a screen refresh rate of 90Hz, a color depth of 3, a first transmission rate of 880*1024*1024 bits, and a second transmission rate of 880*1024*1024 bits, then [(880*1024*1024) / 90] / 3 = r 注视区域 2 Find r 注视区域 =1848, meaning the eye is focused on the target point and the surrounding area within a radius of 1848 pixels, which is defined as the gaze area. [(220*1024*1024) / 90] / 3=r 非注视区域 2 Find r 非注视区域 =924, meaning only 924 pixels are available in the non-focused region. The first transmission rate is obtained by converting the bandwidth of 880M:

[0108] 880M=880*1024K=880*1024*1024b.

[0109] Similarly, based on the bandwidth of 220M, the second transmission rate of 20*1024*1024 is obtained.

[0110] Optionally, a mapping relationship can be pre-established between the first transmission rate, the current screen refresh rate, the color depth, and the display resolution corresponding to the gaze area, and then the display resolution corresponding to the gaze area can be determined based on this mapping relationship. Alternatively, a mapping relationship can be pre-established between the second transmission rate, the current screen refresh rate, the color depth, and the actual resolution corresponding to the non-gaze area, and then the actual resolution corresponding to the non-gaze area can be determined based on this mapping relationship.

[0111] This application determines the display resolution of the content in the gaze area and the actual resolution of the non-gaze area based on a first transmission rate of the gaze area, a second transmission rate of the non-gaze area, the current screen refresh rate, and color depth. This allows the gaze area and the non-gaze area to be divided according to the transmission rate. When the screen refresh rate and resolution are low (i.e., when MIPI has sufficient transmission capacity), the display area of ​​the gaze area is expanded, and the displayed content is clearer. When the screen refresh rate and resolution are high (i.e., when MIPI lacks sufficient transmission capacity), the display area of ​​the gaze area is reduced to meet transmission requirements. While ensuring transmission capacity, the pixels in the gaze area are more densely packed, allowing for clear display of the content in the gaze area.

[0112] Further, step S110 includes the following steps:

[0113] Step S111: According to the pupil position at the moment the image to be displayed is to be shown;

[0114] In this embodiment, distinguishing between areas on the screen that require reduced resolution and those that do not needs to be reduced resolution requires determining the user's pupil position, specifically the pupil position at the moment the image to be displayed is to be shown. The pupil position refers to the current position of the user's pupils on the head-mounted display device. The head-mounted display device can detect the user's real-time pupil position to determine the pupil position at the moment the image to be displayed is to be shown. This pupil position can be the pupil position at the current display moment or the pupil position at an adjacent moment, where the interval between the adjacent moment and the current display moment is less than a preset time. Since the user's pupil position does not change significantly in a short period, the pupil position at an adjacent moment can be used as the pupil position at the current display moment. For example, if the interval between the current moment and the current display moment is less than a preset time, the pupil position detected at the current moment can be used as the pupil position at the moment the image to be displayed is to be shown.

[0115] Step S112: Determine the gaze area in the image to be displayed based on the pupil position.

[0116] In this embodiment, the display time of the image to be displayed can be determined based on the pupil position. The perpendicular line between the center of the user's pupil and the display screen of the head-mounted display device can be used to construct the user's visual cone based on this perpendicular line and the human eye's field of view. Figure 3The first application scenario diagram illustrates this. The optimal field of view for the human eye is generally 60 degrees. Using 60 degrees as the angle between the visual cone section and the aforementioned perpendicular line as the median of this angle, the visual cone is constructed. Based on the pupil position, the overlapping area between the corresponding visual cone and the screen surface of the head-mounted display device, a local high-definition display area on the screen can be determined. This local high-definition display area is the area on the screen that the user can see or clearly perceive. The content displayed within this area retains the best display effect, improving the user's viewing experience. Specifically, the screen surface is determined by the distance between the human eye and the screen. The local high-definition display area is determined by the pupil position at the moment the image to be displayed, the distance between the human eye and the screen, and the optimal field of view for the human eye.

[0117] The content to be displayed in the local high-definition display area is determined, and the area corresponding to this content in the screen is the gaze area. First, the mapping relationship between the screen to be displayed and the screen's display area is established. Based on this mapping relationship and the position of the local high-definition display area on the screen, the gaze area that will be displayed in the local high-definition display area can be determined. Thus, the gaze area in the screen to be displayed can be quickly located by observing the pupil position.

[0118] Further, step S112 includes:

[0119] Step S1121: Determine the difference in relative position between the historical pupil position and the initial pupil position;

[0120] Step S1122: Determine the relative offset and offset direction of the pupil based on the difference in relative positions;

[0121] Step S1123: Determine the pupil position based on the relative offset and the offset direction.

[0122] In this embodiment, the pupil position at the moment the image to be displayed is the pupil position at the moment the image to be displayed is displayed. Since the display moment has not yet occurred, the pupil position at the display moment can be predicted based on the historical pupil position. First, the initial pupil position is determined. The initial pupil position can be the pupil position when the human eye is looking straight ahead, i.e., the center of the eyeball, or it can be the pupil position acquired before acquiring the historical pupil position. The historical pupil position and the actual acquired pupil position can be one or more recently acquired pupil positions, including the current pupil position. The difference between the historical pupil position and the initial position is calculated, thereby calculating the relative offset and offset direction of the pupil relative to the initial position. Based on the relative offset and offset direction, the movement change of the pupil can be fitted, and the pupil position at the display moment can be determined based on the movement change.

[0123] To better understand, a second application scenario is provided:

[0124] When a user puts on a VR headset, the eye-tracking camera activates immediately upon power-on, capturing the interpupillary distance (IPD) of the left and right eyes and automatically adjusting the left and right display centers of the headset. Once adjusted, the initial pupil position (X0, Y0) is recorded and stored in memory. The system continues to track pupil movement, capturing images of the user's eyes at a specific frequency and transmitting them to memory. The current pupil position is then compared with other parameters. Figure 4 The difference between (X1, Y1)(X2, Y2)(X3, Y3)……(Xn, Yn) and the initial position (X0, Y0) is used to calculate the relative offset and direction of the pupil, and determine the pupil position at the moment the image to be displayed is shown.

[0125] By using the difference between the historical pupil position and the initial pupil position, the pupil position at the moment the image to be displayed is predicted, thereby improving the accuracy of determining the gaze area and enhancing the viewing effect of local high-definition displays.

[0126] Optionally, in order to save power consumption, it is also possible to determine whether the pupil position at the time of displaying the image to be displayed will change significantly, thereby determining whether the gaze area will change significantly. If a significant change occurs, the pupil position at the time of displaying the image to be displayed is determined, and the gaze area is then determined. If no significant change occurs, it is not necessary to determine the gaze area based on the pupil position of the image to be displayed, but rather the gaze area of ​​the currently displayed image is used as the gaze area of ​​the image to be displayed.

[0127] Specifically, after step S111, the pupil position offset at the moment the image to be displayed is determined relative to the historical pupil position. If the pupil position offset is greater than the first preset offset, step S112 is executed; otherwise, the gaze area determined at the most recent moment is taken as the gaze area of ​​the image to be displayed, and step S120 is executed.

[0128] Alternatively, before step S111, determine the key points of the historical images in the historical display and the key points of the image to be displayed. These key points can be either the image focus or important elements within the image content. The user's gaze is highly likely to fall on these key points, and the user's gaze is related to the pupil position. For example, when a face appears in the image, the user's pupils will be directed towards the face. If the historical display and the image to be displayed come from the same video, the historical image key points of the historical display and the key points of the image to be displayed can be determined based on moving objects in the video. When the offset of the key points of the image to be displayed relative to the key points of the historical images is greater than a second preset offset, it indicates that the pupil position will change significantly relative to the pupil position at the time the image to be displayed was shown, and step S111 can be executed. Otherwise, the gaze area determined at the most recent moment is used as the gaze area of ​​the image to be displayed.

[0129] Optionally, based on any of the above embodiments, in another embodiment of the control method for the head-mounted display device of the present invention, before step S110 of the control method for the head-mounted display device, the method further includes:

[0130] Receive the video to be played from the video source, and determine the screen to be displayed based on the video to be played;

[0131] The microcontroller unit receives pupil movement position data sent by the pupil capture device, determines the pupil position based on the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device based on the pupil position.

[0132] Receive the local high-definition display area sent by the microcontroller unit;

[0133] The gaze area is determined based on the local high-definition display area;

[0134] The adjusted pixel size image to be displayed is sent to the microcontroller unit, which then controls the screen to display the adjusted pixel size image.

[0135] Reference Figure 6 The head-mounted display device may include a screen (the screen may include a single screen divided into two display areas corresponding to the left and right eyes, or it may include two screens corresponding to the left and right eyes respectively, and be processed in a single-eye corresponding display process), a software processing system, a microcontroller unit connected between the software processing system and the screen, and a pupil capture device connected to the microcontroller unit for acquiring the position of the left and right pupils.

[0136] The control method provided in this embodiment can be applied to a software processing system. The software processing system receives a high-definition video to be played from a video source. Based on the video, the software processing system determines multiple image frames from the video to be played as multiple display frames. The pupil capture device can collect pupil positions at multiple moments and send these multiple pupil positions to the microcontroller unit, or send the pupil position data to the microcontroller unit after each moment's collection. One or more pupil positions sent to the microcontroller unit constitute pupil movement position data. The microcontroller unit receives the pupil movement position data collected by the pupil capture device. Based on the pupil movement position data, the microcontroller unit determines when to display the display frame. The microcontroller determines the display time of the image based on the pupil position at the time the image is to be displayed. It then sends a local high-definition display area to the software processing system. The software processing system determines the gaze area in the image based on the local high-definition display area, and adjusts the current pixel size of the non-gaze area in the image (excluding the gaze area) to the target pixel size, resulting in the image with adjusted pixel size. The software processing system then sends the image to the microcontroller, which controls the screen to display the image with adjusted pixel size.

[0137] This separates the display task from the process of processing the display image, which reduces the CPU computational load on the screen display and reduces the power consumption of the screen display. This allows the computing power and power consumption of the head-mounted display device to be used more effectively, improving the effect of local high-definition display.

[0138] The embodiments of the present invention provide an embodiment of an image display method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0139] Based on the same inventive concept, this application also provides a computer-readable storage medium storing an image display program. When the image display program is executed by a processor, it implements the various steps of the image display method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0140] Since the storage medium provided in this application embodiment is the storage medium used to implement the method of this application embodiment, those skilled in the art can understand the specific structure and variations of the storage medium based on the method described in this application embodiment, and therefore will not be repeated here. All storage media used in the method of this application embodiment are within the scope of protection of this application.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0142] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, television, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0144] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image display method, characterized in that, The image display method, applied to head-mounted display devices, includes: Determine the gaze area and non-gaze area in the image to be displayed, wherein the non-gaze area is the area in the image to be displayed other than the gaze area; The initial display resolution corresponding to the non-focused area is determined based on the preset resolution of the display screen and the display resolution corresponding to the gaze area. The pixel magnification ratio of the non-focused area is determined based on the actual resolution corresponding to the non-focused area and the initial display resolution, wherein the actual resolution corresponding to the focused area is greater than the actual resolution corresponding to the non-focused area. Based on the pixel magnification ratio, the current pixel size of the non-focused region is adjusted to the target pixel size, where the target pixel size is larger than the current pixel size; Control the head-mounted display device to display the image to be displayed after adjusting the pixel size; Determining the gaze area in the image to be displayed includes: Determine the difference in relative position between the historical pupil position and the initial pupil position; The relative offset and direction of the pupil are determined based on the difference in relative position. The pupil position is determined based on the relative offset and the offset direction; The gaze area in the image to be displayed is determined based on the pupil position; Before the step of determining the initial display resolution corresponding to the non-focused area based on the preset resolution of the display screen and the display resolution corresponding to the gaze area, the method further includes: Get the current system's maximum transmission rate; Determine the transmission rate ratio corresponding to the gaze region and the non-gaze region; Based on the maximum transmission rate and the transmission rate percentage, a first transmission rate corresponding to the gaze region and a second transmission rate corresponding to the non-gaze region are determined, wherein the first transmission rate is greater than the second transmission rate. Determine the current screen refresh rate and color depth; Based on the first transmission rate, the current screen refresh rate, and the color depth, the display resolution corresponding to the gaze area is determined, and based on the second transmission rate, the current screen refresh rate, and the color depth, the actual resolution corresponding to the non-gaze area is determined.

2. The image display method as described in claim 1, characterized in that, The steps of determining the display resolution corresponding to the gaze area based on the first transmission rate, the current screen refresh rate, and the color depth, and determining the actual resolution corresponding to the non-gaze area based on the second transmission rate, the current screen refresh rate, and the color depth, include: A first resolution is determined based on the ratio of the first transmission rate to the current screen refresh rate. A second resolution is determined based on the ratio of the first resolution to the color depth. The second resolution is then square-rooted to obtain the display resolution corresponding to the gaze area. Based on the ratio of the second transmission rate to the current screen refresh rate, a third resolution is determined. Based on the ratio of the third resolution to the color depth, a fourth resolution is determined. The square root of the fourth resolution is then performed to obtain the actual resolution corresponding to the non-focused area.

3. The image display method as described in claim 1, characterized in that, The step of determining the initial display resolution corresponding to the non-focused area based on the preset resolution of the display screen and the display resolution corresponding to the gaze area includes: The difference between the preset resolution of the display screen and the display resolution corresponding to the gaze area is determined as the initial display resolution corresponding to the non-gaze area.

4. The image display method as described in claim 1, characterized in that, The step of determining the pixel magnification ratio corresponding to the non-focused area based on the actual resolution corresponding to the non-focused area and the initial display resolution includes: The ratio of the actual resolution corresponding to the non-focused area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-focused area.

5. A head-mounted display device, characterized in that, The head-mounted display device includes: a memory, a processor, and an image display program stored in the memory and executable on the processor, wherein the image display program, when executed by the processor, implements the steps of the image display method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image display program, which, when executed by a processor, implements the steps of the image display method according to any one of claims 1-4.

Images