Image display method and device and computer readable storage medium

Abstract

The invention discloses an image display method and device and a computer readable storage medium, and the method comprises the steps: determining a fixation region and a non-fixation region in a to-be-displayed image, the non-fixation region being a region except the fixation region in the to-be-displayed image; determining an initial display resolution corresponding to the non-gazing area according to a preset resolution of a display screen and a display resolution corresponding to the gazing area; according to the actual resolution corresponding to the non-gazing area and the initial display resolution, determining a pixel magnification ratio corresponding to the non-gazing area, the actual resolution corresponding to the gazing area being greater than the actual resolution corresponding to the non-gazing area; adjusting the current pixel size of the non-fixation area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is greater than the current pixel size; and controlling the head-mounted display device to display the to-be-displayed picture after the pixel size is adjusted, thereby improving the display effect of the head-mounted display device.

Description

Technical Field

[0001] The present invention relates to the technical field of head mounted display devices, and in particular to an image display method, device and computer readable storage medium. Background Art

[0002] With the development of image processing technology, people have higher and higher requirements for the display effects of head-mounted display devices. In order to meet the needs of users, manufacturers have begun to pursue higher display effects. In related technologies, the captured image is rendered and the rendered image is transmitted to the display screen for display. However, when the resolution of the image is high, the amount of image data transmitted increases and the transmission rate is slow, which leads to a poor display effect of the head-mounted display device. Summary of the invention

[0003] The embodiments of the present application provide an image display method, device, and computer-readable storage medium to improve the display effect of a head-mounted display device.

[0004] The present application provides an image display method, which is applied to a head mounted display device. The image display method includes:

[0005] Determine a fixation area and a non-fixation area in the picture to be displayed, wherein the non-fixation area is an area of ​​the picture to be displayed except the fixation area;

[0006] Determining an initial display resolution corresponding to the non-attention area according to a preset resolution of the display screen and a display resolution corresponding to the attention area;

[0007] Determining a pixel magnification ratio corresponding to the non-attention area according to an actual resolution corresponding to the non-attention area and the initial display resolution, wherein the actual resolution corresponding to the attention area is greater than the actual resolution corresponding to the non-attention area;

[0008] Adjusting a current pixel size of the non-attention area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is larger than the current pixel size;

[0009] Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.

[0010] Optionally, before the step of determining the initial display resolution corresponding to the non-attention area according to the preset resolution of the display screen and the display resolution corresponding to the attention area, the step further includes:

[0011] Determine a first transmission rate of the attention area and a second transmission rate of the non-attention area, wherein the first transmission rate is greater than the second transmission rate;

[0012] Determine the current screen refresh rate and color depth;

[0013] A display resolution corresponding to the gaze area is determined according to the first transmission rate, the current screen refresh rate, and the color depth, and an actual resolution corresponding to the non-gaze area is determined according to the second transmission rate, the current screen refresh rate, and the color depth.

[0014] Optionally, the step of determining a display resolution corresponding to the gaze area according to the first transmission rate, the current screen refresh rate, and the color depth, and determining an actual resolution corresponding to the non-gaze area according to the second transmission rate, the current screen refresh rate, and the color depth includes:

[0015] Determine a first resolution according to a ratio of the first transmission rate to the current screen refresh rate, determine a second resolution according to a ratio of the first resolution to the color depth, and perform a square root operation on the second resolution to obtain a display resolution corresponding to the gaze area;

[0016] and, determining a third resolution according to a ratio of the second transmission rate to the current screen refresh rate, determining a fourth resolution according to a ratio of the third resolution to the color depth, performing a square root operation on the fourth resolution, and obtaining an actual resolution corresponding to the non-attention area.

[0017] Optionally, the step of determining the first transmission rate of the attention area and the second transmission rate of the non-attention area includes:

[0018] Get the maximum transmission rate of the current system;

[0019] Determine the transmission rate ratios corresponding to the attention area and the non-attention area;

[0020] A first transmission rate corresponding to the attention area and a second transmission rate corresponding to the non-attention area are determined according to the maximum transmission rate and the transmission rate ratio.

[0021] Optionally, the step of determining the initial display resolution corresponding to the non-attention area according to the preset resolution of the display screen and the display resolution corresponding to the attention area includes:

[0022] A difference between a preset resolution of the display screen and a display resolution corresponding to the gaze area is determined as an initial display resolution corresponding to the non-gaze area.

[0023] Optionally, the step of determining the pixel magnification ratio corresponding to the non-attention area according to the actual resolution corresponding to the non-attention area and the initial display resolution includes:

[0024] The ratio of the actual resolution corresponding to the non-attention area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-attention area.

[0025] Optionally, determining a gaze area in the picture to be displayed includes:

[0026] Determine the pupil position at the time when the picture to be displayed is displayed;

[0027] A gaze area in the picture to be displayed is determined according to the pupil position.

[0028] Optionally, the step of determining the pupil position at the time when the picture to be displayed is displayed includes:

[0029] determining the difference in relative position between the historical pupil position and the initial pupil position;

[0030] Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions;

[0031] The pupil position is determined according to the relative offset and the offset direction.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a head-mounted display device, which includes: a memory, a processor, and an image display program stored in the memory and executable on the processor, and the image display program implements the steps of the above-mentioned image display method when executed by the processor.

[0033] In addition, to achieve the above-mentioned purpose, the present invention further provides a storage medium on which an image display program is stored, and when the image display program is executed by a processor, the steps of the above-mentioned image display method are implemented.

[0034] A technical solution of an image display method, device and computer-readable storage medium provided in an embodiment of the present application, the method divides a picture to be displayed into a gaze area that requires high-definition display and a non-gaze area that does not require high-definition display, and determines an initial display resolution corresponding to the non-gaze area according to a preset resolution of the display screen and a display resolution corresponding to the gaze area, the gaze area is displayed with an actual resolution, the current pixel size of the non-gaze area is adjusted to a target pixel size with a larger pixel size, the non-gaze area is displayed blurred, the data transmission amount of the non-gaze area is reduced, the data transmission efficiency of the non-gaze area is improved, and the display effect of the head-mounted display device is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a head mounted display device according to an embodiment of the present invention;

[0036] Figure 2It is a flowchart of a first embodiment of an image display method of the present invention;

[0037] Figure 3 This is a schematic diagram of a first application scenario involved in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of a second application scenario involved in an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of fixation area and non-fixation area;

[0040] Figure 6 FIG. 1 is a schematic diagram of an embodiment of the present invention.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire invention. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0043] In the related art, the captured image is rendered and the rendered image is transmitted to the display screen for display. When the resolution of the image is high, the image data transmission volume increases and the transmission rate is slow, which leads to a poor display effect of the head mounted display device.

[0044] In order to improve the display effect of a head-mounted display device, an embodiment of the present invention provides a control method for a head-mounted display device, wherein the main steps of the method include: determining a gaze area and a non-gaze area in a picture to be displayed, wherein the non-gaze area is an area in the picture to be displayed other than the gaze area; determining an initial display resolution corresponding to the non-gaze area according to a preset resolution of a display screen and a display resolution corresponding to the gaze area; determining a pixel magnification ratio corresponding to the non-gaze area according to an actual resolution corresponding to the non-gaze area and the initial display resolution, wherein the actual resolution corresponding to the gaze area is greater than the actual resolution corresponding to the non-gaze area; adjusting a current pixel size of the non-gaze area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is greater than the current pixel size; and controlling the head-mounted display device to display the picture to be displayed after the pixel size is adjusted. The method divides a picture to be displayed into a fixation area that requires high-definition display and a non-fixation area that does not require high-definition display, and determines the initial display resolution corresponding to the non-fixation area according to the preset resolution of the display screen and the display resolution corresponding to the fixation area. The fixation area is displayed using the actual resolution, and the current pixel size of the non-fixation area is adjusted to a target pixel size with a larger pixel size, and the non-fixation area is displayed blurred, thereby reducing the data transmission volume in the non-fixation area and improving the data transmission efficiency in the non-fixation area, thereby improving the display effect of the head-mounted display device.

[0045] In addition, the present application determines the display resolution of the display content corresponding to the gaze area and the actual resolution corresponding to the non-gaze area according to the first transmission rate of the gaze area, the second transmission rate of the non-gaze area, the current screen refresh rate and the color depth. The area range of the gaze area and the non-gaze area can be divided according to the transmission rate, so that when the screen refresh rate is low and the resolution is low, that is, when MIPI has sufficient transmission capacity, the display range of the gaze area is expanded and the displayed content is clearer. When the screen refresh rate is high and the resolution is high, that is, MIPI does not have sufficient transmission capacity, in order to meet the transmission capacity, the display range of the gaze area is reduced. While ensuring the transmission capacity, the pixels in the gaze area are denser, and the content of the gaze area can be clearly displayed.

[0046] The content to be protected by the claims of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0048] The terminal in the embodiment of the present invention may be a head mounted display device.

[0049] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The memory 1003 may be a high-speed RAM memory or a stable memory, such as a disk memory. The memory 1003 may optionally be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] like Figure 1 As shown, the memory 1003 as a computer storage medium may include an operating system and a control program of the head mounted display device.

[0052] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the control program of the head mounted display device stored in the memory 1003, and perform the following operations:

[0053] Determine a fixation area and a non-fixation area in the picture to be displayed, wherein the non-fixation area is an area of ​​the picture to be displayed except the fixation area;

[0054] Determining an initial display resolution corresponding to the non-attention area according to a preset resolution of the display screen and a display resolution corresponding to the attention area;

[0055] Determining a pixel magnification ratio corresponding to the non-attention area according to an actual resolution corresponding to the non-attention area and the initial display resolution, wherein the actual resolution corresponding to the attention area is greater than the actual resolution corresponding to the non-attention area;

[0056] Adjusting a current pixel size of the non-attention area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is larger than the current pixel size;

[0057] Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.

[0058] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0059] Determine a first transmission rate of the attention area and a second transmission rate of the non-attention area, wherein the first transmission rate is greater than the second transmission rate;

[0060] Determine the current screen refresh rate and color depth;

[0061] A display resolution corresponding to the gaze area is determined according to the first transmission rate, the current screen refresh rate, and the color depth, and an actual resolution corresponding to the non-gaze area is determined according to the second transmission rate, the current screen refresh rate, and the color depth.

[0062] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0063] Determine a first resolution according to a ratio of the first transmission rate to the current screen refresh rate, determine a second resolution according to a ratio of the first resolution to the color depth, and perform a square root operation on the second resolution to obtain a display resolution corresponding to the gaze area;

[0064] and, determining a third resolution according to a ratio of the second transmission rate to the current screen refresh rate, determining a fourth resolution according to a ratio of the third resolution to the color depth, performing a square root operation on the fourth resolution, and obtaining an actual resolution corresponding to the non-attention area.

[0065] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0066] Get the maximum transmission rate of the current system;

[0067] Determine the transmission rate ratios corresponding to the attention area and the non-attention area;

[0068] A first transmission rate corresponding to the attention area and a second transmission rate corresponding to the non-attention area are determined according to the maximum transmission rate and the transmission rate ratio.

[0069] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0070] A difference between a preset resolution of the display screen and a display resolution corresponding to the gaze area is determined as an initial display resolution corresponding to the non-gaze area.

[0071] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0072] The ratio of the actual resolution corresponding to the non-attention area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-attention area.

[0073] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0074] Determine the pupil position at the time when the picture to be displayed is displayed;

[0075] A gaze area in the picture to be displayed is determined according to the pupil position.

[0076] The processor 1001 may be used to call a control program of the head mounted display device stored in the memory 1003 and perform the following operations:

[0077] determining the difference in relative position between the historical pupil position and the initial pupil position;

[0078] Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions;

[0079] The pupil position is determined according to the relative offset and the offset direction.

[0080] like Figure 2 As shown, in the first embodiment of the present application, the image display method of the present application includes the following steps:

[0081] Step S110, determining a fixation area and a non-fixation area in the picture to be displayed, wherein the non-fixation area is an area of ​​the picture to be displayed except the fixation area.

[0082] In this embodiment, the head-mounted display device refers to a display device worn on the user's head, such as a virtual display device, an enhanced display device, etc. The user watches the content displayed on the screen of the head-mounted display device at a close distance. Due to the close distance, the field of view that the user can see is small and may not completely include the screen of the head-mounted display device. Therefore, the local high-definition display can reduce the resolution of the display content that the user cannot see or cannot see clearly, and retain the display effect of the display content that the user can see or focus on. In this way, the power consumption of the product can be reduced without changing the viewing effect of the user.

[0083] In this embodiment, the gaze area is the position of the screen where the eyeball looks at the screen at the moment of display of the picture to be displayed, including the center of the looking position and some surrounding pixels, and the number of surrounding pixels is determined by the screen refresh rate and resolution, that is, it is related to the transmission capacity of MIPI. When the screen refresh rate is low and the resolution is low, that is, when MIPI has sufficient transmission capacity, the display range of the surrounding pixels is expanded. When the screen refresh rate is high and the resolution is high, that is, MIPI does not have sufficient transmission capacity, in order to meet the transmission capacity, the display range of the surrounding pixels is reduced. The non-gaze area refers to other areas in the picture to be displayed except the gaze area, that is, the direct position where the eyeball looks at the screen, that is, the pixel area other than the center and some surrounding pixels of the looking position.

[0084] As reference Figure 5 According to the human eye field angle and pupil position, the gaze area that the user can see is determined as the square area Q, and the area other than the square area Q in the image to be displayed is determined as the non-gaze area.

[0085] Step S120: determining an initial display resolution corresponding to the non-attention area according to a preset resolution of the display screen and a display resolution corresponding to the attention area.

[0086] In this embodiment, the preset resolutions corresponding to display screens of different models are different. The preset resolutions may be fixed or may be pre-set in multiples, and the appropriate resolutions may be selected according to different application scenarios. For example, the preset resolution of the current display screen is selected as 4000*4000. The display resolution corresponding to the gaze area refers to the display resolution of the display content in the gaze area. The display resolution corresponding to the gaze area may be calculated based on the first transmission rate, the current screen refresh rate, and the color depth of the gaze area, so that when the screen refresh rate is low and the resolution is low, that is, when MIPI has sufficient transmission capacity, the display range of the gaze area is expanded and the displayed content is clearer. When the screen refresh rate is high and the resolution is high, that is, MIPI does not have sufficient transmission capacity, in order to meet the transmission capacity, the display range of the gaze area is reduced. While ensuring the transmission capacity, the pixels of the gaze area are denser, and the content of the gaze area can be clearly displayed.

[0087] In this embodiment, for the display content of the gaze area, more resources are occupied and more image data is rendered as much as possible. This application sets the correspondence between the display pixels and the actual pixels of the display content corresponding to the gaze area to 1:1, and sets the display pixels and the actual pixels of the display content to be the same, that is, the actual resolution corresponding to the gaze area is used as the display resolution of the display content corresponding to the gaze area, and the display content corresponding to the gaze area is restored one-to-one, and the display content corresponding to the gaze area is not compressed, so that the display content in the user's gaze area can be displayed specifically and clearly. Optionally, the difference between the preset resolution of the display screen and the display resolution corresponding to the gaze area can be determined as the initial display resolution corresponding to the non-gaze area, thereby determining the number of pixels required to be displayed in the non-gaze area. For example, the preset resolution of the current display screen is 4000*4000, and the display resolution corresponding to the gaze area is 1848, then the initial display resolution corresponding to the non-gaze area is 4000-1848=2152.

[0088] Step S130, determining a pixel magnification ratio corresponding to the non-attention area according to an actual resolution corresponding to the non-attention area and the initial display resolution, wherein the actual resolution corresponding to the attention area is greater than the actual resolution corresponding to the non-attention area.

[0089] In this embodiment, the actual resolution corresponding to the non-attention area can be calculated according to the second transmission rate of the non-attention area, the current screen refresh rate and the color depth. This application sets the correspondence between the display pixels and the actual pixels of the display content corresponding to the non-attention area to N:1, where N is the pixel magnification ratio, and N is greater than or equal to 2.

[0090] Optionally, the ratio of the actual resolution corresponding to the non-attention area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-attention area. For example, assuming that the actual resolution corresponding to the non-attention area is calculated to be 924 according to the second transmission rate of the non-attention area, the current screen refresh rate and the color depth, when the initial display resolution corresponding to the non-attention area is 2152, then the pixel magnification ratio corresponding to the non-attention area is N=2152 / 924=2.3, and N=3 is rounded, which means that the three surrounding adjacent pixels share the same display data. Since the display content of the non-attention area occupies less resources and reduces the rendering of data, the pixels in the area that the eyes do not pay attention to are set to N:1 display with the actual pixels, that is, the content of the central pixel is displayed within the range of N pixels around a pixel, which reduces the data for image rendering, thereby reducing the amount of data transmission and improving the display effect of the image.

[0091] Step S140: adjusting the current pixel size of the non-attention area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is larger than the current pixel size.

[0092] In this embodiment, a picture in a non-attention area of ​​a picture to be displayed is determined according to the attention area. The picture to be displayed is composed of an arrangement of pixels. The pixel size of the pixels in the picture to be displayed is the current pixel size. The current pixel size is adjusted to a target pixel size. The target pixel size is larger than the current pixel size. After adjusting to the target pixel size, the number of pixels per unit area of ​​the non-attention area becomes smaller, so that the clarity in the non-attention area is reduced, and the corresponding power consumption during display is reduced.

[0093] Optionally, determine the attention area occupied by the target pixel points in the non-attention area after adjusting to the target pixel size, merge the current pixel points in the non-attention area within the attention area into the target pixel points, and when merging, determine the target pixel value of the target pixel point based on the pixel value corresponding to the current pixel point in the non-attention area within the attention area. For example, take the average of the pixel values ​​corresponding to the current pixel points in the non-attention area within the attention area as the target pixel value, and construct a target picture corresponding to the non-attention area based on the target pixel value and the attention area.

[0094] Step S150, controlling the head mounted display device to display the image to be displayed after the pixel size is adjusted.

[0095] In this embodiment, after the pixel size of the non-attention area is adjusted to the target pixel size, a target image of the non-attention area is generated accordingly, and the target image obtained after the pixel size is adjusted and the image corresponding to the attention area whose pixel size is not adjusted are combined into an image to be displayed after the pixel size is adjusted.

[0096] Optionally, the image of the non-attention area and the image of the attention area where the target pixel is adjusted are rendered separately, and the rendered images are spliced ​​to obtain a spliced ​​picture to be displayed. The spliced ​​picture to be displayed is transmitted to the screen via MIPI for display.

[0097] Since the current pixel size of the non-attention area is adjusted to a larger target pixel size, the number of pixels per unit area of ​​the non-attention area becomes smaller, which reduces the clarity in the non-attention area and reduces the corresponding display power consumption. The pixel size of the attention area that can be viewed or seen clearly by the user at the display moment is still the current pixel size, and the clarity has not changed. The clarity will be higher than that of the non-attention area, so that the user's viewing effect will not be reduced, thereby achieving local high-definition display.

[0098] According to the above technical solution, this embodiment divides the picture to be displayed into a gaze area that requires high-definition display and a non-gaze area that does not require high-definition display, and determines the initial display resolution corresponding to the non-gaze area according to the preset resolution of the display screen and the display resolution corresponding to the gaze area. The gaze area is displayed with the actual resolution, and the current pixel size of the non-gaze area is adjusted to a target pixel size with a larger pixel size, and the non-gaze area is displayed blurred, thereby reducing the data transmission amount in the non-gaze area, improving the data transmission efficiency in the non-gaze area, and improving the display effect of the head-mounted display device.

[0099] Further, before step S120, the method includes:

[0100] Step S210: determining a first transmission rate of the attention area and a second transmission rate of the non-attention area, wherein the first transmission rate is greater than the second transmission rate.

[0101] In this embodiment, in order to ensure stable transmission, the first transmission rate and the second transmission rate are different. Since the clarity required in the gaze area is higher than that in the non-gaze area, in order to improve the image display effect, the first transmission rate corresponding to the gaze area should be set to be greater than the second transmission rate corresponding to the non-gaze area. The first transmission rate and the second transmission rate can be determined according to the maximum transmission rate of the current system and the proportion of the transmission rates corresponding to the gaze area and the non-gaze area. In this way, when the screen refresh rate is low and the resolution is low, that is, when MIPI has sufficient transmission capacity, the display range of the gaze area is expanded and the displayed content is clearer. When the screen refresh rate is high and the resolution is high, that is, MIPI does not have sufficient transmission capacity, in order to meet the transmission capacity, the display range of the gaze area is reduced. While ensuring the transmission capacity, the pixels in the gaze area are denser, and the content of the gaze area can be clearly displayed. Optionally, the first transmission rate and the second transmission rate can also be preset according to actual conditions.

[0102] Optionally, obtain the maximum transmission rate of the current system; determine the transmission rate ratio corresponding to the gaze area and the non-attention area; and determine the first transmission rate corresponding to the gaze area and the second transmission rate corresponding to the non-attention area according to the maximum transmission rate and the transmission rate ratio. Assuming that the maximum transmission rate of MIPI single LANE is 1.5G, in order to ensure stable transmission, the actual maximum available transmission rate is 1.1G, then 80%, or 880M bandwidth, is allocated to data transmission in the gaze area, and the remaining 20%, or 220M bandwidth, is allocated to data transmission in the non-attention area.

[0103] Step S220, determining the current screen refresh rate and color depth.

[0104] In this embodiment, the current screen refresh rate and color depth can be preset or fixed according to the situation, for example, the current screen refresh rate is set to 90 Hz and the color depth is set to 3.

[0105] Step S230: determining a display resolution corresponding to the gaze area according to the first transmission rate, the current screen refresh rate, and the color depth, and determining an actual resolution corresponding to the non-gaze area according to the second transmission rate, the current screen refresh rate, and the color depth.

[0106] Optionally, a first resolution is determined based on the ratio of the first transmission rate and the current screen refresh rate, a second resolution is determined based on the ratio of the first resolution and the color depth, and the second resolution is squared to obtain a display resolution corresponding to the gaze area; and a third resolution is determined based on the ratio of the second transmission rate and the current screen refresh rate, a fourth resolution is determined based on the ratio of the third resolution and the color depth, and the fourth resolution is squared to obtain an actual resolution corresponding to the non-gaze area.

[0107] Assuming the screen refresh rate is 90Hz, the color depth is 3, the first transmission rate is 880*1024*1024b, and the second transmission rate is 880*1024*1024, then [(880*1024*1024) / 90] / 3=r 注视区域 2 , find r 注视区域 =1848, that is, the target point and the surrounding radius of 1848 pixels are set as the gaze area. [(220*1024*1024) / 90] / 3=r 非注视区域 2 , find r 非注视区域 =924, that is, the available pixels in the non-attention area are only 924. Among them, the first transmission rate is obtained by converting the bandwidth 880M:

[0108] 880M=880*1024K=880*1024*1024b.

[0109] Similarly, the second transmission rate 20*1024*1024 is obtained according to the bandwidth 220M.

[0110] Optionally, a mapping relationship between the first transmission rate, the current screen refresh rate, the color depth and the display resolution corresponding to the gaze area may be pre-established, and the display resolution corresponding to the gaze area may be subsequently determined based on the mapping relationship. A mapping relationship between the second transmission rate, the current screen refresh rate, the color depth and the actual resolution corresponding to the non-gaze area may also be pre-established, and the actual resolution corresponding to the non-gaze area may be subsequently determined based on the mapping relationship.

[0111] This application determines the display resolution of the display content corresponding to the gaze area and the actual resolution corresponding to the non-gaze area according to the first transmission rate of the gaze area, the second transmission rate of the non-gaze area, the current screen refresh rate and the color depth. The area range of the gaze area and the non-gaze area can be divided according to the transmission rate, so that when the screen refresh rate is low and the resolution is low, that is, when MIPI has sufficient transmission capacity, the display range of the gaze area is expanded and the displayed content is clearer. When the screen refresh rate is high and the resolution is high, that is, MIPI does not have sufficient transmission capacity, in order to meet the transmission capacity, the display range of the gaze area is reduced. While ensuring the transmission capacity, the pixels in the gaze area are denser, and the content of the gaze area can be clearly displayed.

[0112] Furthermore, the step S110 includes the following steps:

[0113] Step S111, according to the pupil position at the time of displaying the picture to be displayed;

[0114] In this embodiment, the areas on the screen that need to reduce the definition and those that do not need to reduce the definition need to be determined based on the pupil position of the user, and the pupil position at the time when the picture to be displayed is displayed needs to be determined. The pupil position is the position of the current user's pupil of the head-mounted display device, and the head-mounted display device can detect the user's real-time pupil position, and then determine the pupil position at the time when the picture to be displayed is displayed. The pupil position at the time when the picture to be displayed is displayed can be the pupil position at the display time, and the pupil position at the adjacent time of the display time can also be used as the pupil position at the display time, wherein the interval time between the adjacent time and the display time is less than the preset time. Since the user's pupil position will not change too much in a short time, the pupil position at the adjacent time can be used as the pupil position at the display time. For example, if the interval time between the current time and the display time of the picture to be displayed is less than the preset time, the pupil position detected at the current time can be used as the pupil position at the time when the picture to be displayed is displayed.

[0115] Step S112, determining a gaze area in the image to be displayed according to the pupil position.

[0116] In this embodiment, the time when the image to be displayed is displayed can be determined according to the pupil position. The vertical line between the center of the user's pupil and the display screen of the head-mounted display device can be used to construct the user's visual cone according to the vertical line and the human eye field angle. Figure 3The schematic diagram of the first application scenario shows that the best viewing angle of the human eye is generally 60 degrees. The viewing cone is constructed with 60 degrees as the angle of the cross section of the viewing cone, and the median line of the angle is the vertical line mentioned above. According to the overlapping area between the pupil position, the corresponding viewing cone and the surface where the screen of the head-mounted display device is located, the local high-definition display area on the screen can be determined. The local high-definition display area is also the display area on the screen that the user can see or see clearly. The display content in the local high-definition display area can retain the best display effect to improve the user's viewing effect. Specifically, the surface where the screen is located can be determined according to the distance between the human eye and the screen, and the local high-definition display area of ​​the screen can be determined according to the pupil position at the time of display of the picture to be displayed, the distance between the human eye and the screen, and the best viewing angle of the human eye.

[0117] Determine the display content of the picture to be displayed in the local high-definition display area, and the area in the picture to be displayed corresponding to the display content is the gaze area. First, determine the mapping relationship between the picture to be displayed and the display area of ​​the screen. According to the mapping relationship and the position of the local high-definition display area of ​​the screen on the screen, the gaze area in the picture to be displayed that will be displayed in the local high-definition display area can be determined. In this way, the gaze area in the picture to be displayed can be quickly located by the pupil position.

[0118] Furthermore, the step S112 includes:

[0119] Step S1121, determining the relative position difference between the historical pupil position and the initial pupil position;

[0120] Step S1122, determining the relative offset and offset direction of the pupil according to the difference in the relative positions;

[0121] Step S1123: determining the pupil position according to the relative offset and the offset direction.

[0122] In this embodiment, the pupil position at the moment when the picture to be displayed is displayed may be the pupil position at the moment when the picture to be displayed is displayed. Since the display moment has not occurred, the pupil position at the moment of display may be predicted based on the historical pupil position. First, the initial pupil position is determined. The initial pupil position may be the pupil position of the human eye looking straight ahead, that is, the center position of the eyeball, or the pupil position collected before the historical pupil position is collected. The historical pupil position and the actually collected pupil position may be one or more pupil positions collected the latest including the current pupil position, including the pupil position collected at the current moment. The difference between the relative position of the historical pupil position and the initial position is calculated, thereby calculating the relative offset and offset direction of the pupil relative to the initial position. The movement change of the pupil may be fitted based on the relative offset and offset direction, and the pupil position at the moment of display may be determined based on the movement change.

[0123] For a better understanding, the second application scenario is provided:

[0124] When the user wears the VR headset, the eye tracking camera is turned on immediately to capture the left and right pupil distances and automatically adjust the left and right display centers of the headset. After the adjustment is complete, the eye center position (X0, Y0) is recorded as the initial pupil position and recorded in the memory. The movement of the pupil is tracked, and the human eye image is captured at a certain frequency and transmitted to the memory. The current pupil position is compared with the current pupil position. Figure 4 The difference between (X1, Y1)(X2, Y2)(X3, Y3)...(Xn, Yn) and the initial position (X0, Y0) is used to calculate the relative offset and offset direction of the pupil, and determine the pupil position at the moment when the image to be displayed is displayed.

[0125] In this way, the pupil position at the moment of displaying the image to be displayed is predicted through the difference in relative position between the historical pupil position and the initial pupil position, thereby improving the accuracy of determining the gaze area and improving the viewing effect of the local high-definition display.

[0126] Optionally, in order to save power consumption, it is also possible to determine whether the pupil position at the time the picture to be displayed changes significantly, thereby determining whether the gaze area changes significantly. When a significant change occurs, the pupil position at the time the picture to be displayed is determined, and then the gaze area is determined. When no significant change is sent, there is no need to determine the gaze area based on the pupil position of the picture to be displayed, but the gaze area of ​​the currently displayed picture is used as the gaze area of ​​the picture to be displayed.

[0127] Specifically, after step S111, the pupil position offset of the pupil position at the time of display of the image to be displayed relative to the historical pupil position is determined. When the pupil position offset is greater than the first preset offset, step S112 is executed; otherwise, the gaze area determined at the most recent moment is used as the gaze area of ​​the image to be displayed, and step S120 is executed.

[0128] Or before step S111, determine the historical image key points of the historical display screen, and determine the image key points of the screen to be displayed. The image key points can be the focus of the image or the focus of the image content. The user's gaze point is likely to fall on the image key points, and the user's gaze point is related to the pupil position. For example, when a face appears in the image, the user's pupil will be oriented toward the face position. If the historical display screen and the screen to be displayed are from the same video, then the historical image key points of the historical display screen and the image key points of the screen to be displayed can be determined based on the moving objects in the video. When the offset of the image key points of the screen to be displayed relative to the historical image key points is greater than the second preset offset, it indicates that the pupil position will change significantly when the screen to be displayed is displayed relative to the pupil position at the time when the historical display image is displayed, and step S111 can be executed, otherwise the gaze area determined at the most recent moment is used as the gaze area of ​​the screen to be displayed.

[0129] Optionally, based on any of the above embodiments, in another embodiment of the control method of a head mounted display device of the present invention, before step S110, the control method of the head mounted display device further includes:

[0130] Receiving a video to be played sent by a video source, and determining the picture to be displayed according to the video to be played;

[0131] The micro-control unit receives the pupil movement position data sent by the pupil capture device, determines the pupil position according to the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device according to the pupil position;

[0132] Receiving the local high-definition display area sent by the micro-control unit;

[0133] Determining the gaze area according to the local high-definition display area;

[0134] The picture to be displayed after the pixel size is adjusted is sent to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.

[0135] Reference Figure 6 The head-mounted display device may include a screen (the screen may include a screen divided into two display areas corresponding to the left and right eyes, or may include two screens corresponding to the left and right eyes respectively, and processed with the display process corresponding to a single eye), a software processing system, a micro-control unit connected between the software processing system and the screen, and a pupil capture device connected to the micro-control unit and used to collect the pupil positions of the left and right eyes.

[0136] The control method provided in this embodiment can be applied to a software processing system. The software processing system receives a high-definition video to be played sent by a video source. The software processing system determines multiple image frames of the video to be played as multiple pictures to be displayed according to the video to be played. The pupil capture device can collect pupil positions at multiple moments and send the pupil positions at multiple moments to the micro-control unit or send the pupil positions to the micro-control unit every time a pupil position is collected. One or more pupil positions sent to the micro-control unit constitute pupil movement position data. The micro-control unit receives the pupil movement position data collected by the pupil capture device. The micro-control unit determines the display time of the picture to be displayed according to the pupil movement position data. The pupil position at that moment is determined, and then the micro-control unit determines the display time of the picture to be displayed according to the pupil position at that moment, and the local high-definition display area on the screen, and the micro-control unit sends the local high-definition display area to the software processing system; the software processing system determines the gaze area in the picture to be displayed according to the local high-definition display area, and adjusts the current pixel size corresponding to the non-gaze area of ​​the picture to be displayed except the gaze area to the target pixel size according to the gaze area in the picture to be displayed, and obtains the picture to be displayed after the pixel size is adjusted, and the software processing system sends the picture to be displayed to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.

[0137] This way, the display tasks and the display screen processing processes are separated, which can reduce the CPU computing power loss of the screen display part and the power consumption of the screen display part, so that the computing power and power consumption of the head-mounted display device can be used more effectively and improve the local high-definition display effect.

[0138] The embodiment of the present invention provides an embodiment of the image display method. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0139] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores an image display program. When the image display program is executed by a processor, it implements the various steps of the image display method described above and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] Since the storage medium provided in the embodiments of the present application is the storage medium used to implement the method of the embodiments of the present application, based on the method introduced in the embodiments of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the methods of the embodiments of the present application belong to the scope of protection of this application.

[0141] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0142] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0144] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image display method, characterized in that: Applied to a head mounted display device, the image display method comprises: Determine a fixation area and a non-fixation area in the picture to be displayed, wherein the non-fixation area is an area of ​​the picture to be displayed except the fixation area; Determining an initial display resolution corresponding to the non-attention area according to a preset resolution of the display screen and a display resolution corresponding to the attention area; Determining a pixel magnification ratio corresponding to the non-attention area according to an actual resolution corresponding to the non-attention area and the initial display resolution, wherein the actual resolution corresponding to the attention area is greater than the actual resolution corresponding to the non-attention area; Adjusting a current pixel size of the non-attention area to a target pixel size based on the pixel magnification ratio, wherein the target pixel size is larger than the current pixel size; Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.

2. The image display method according to claim 1, wherein: Before the step of determining the initial display resolution corresponding to the non-attention area according to the preset resolution of the display screen and the display resolution corresponding to the attention area, the method further includes: Determine a first transmission rate of the attention area and a second transmission rate of the non-attention area, wherein the first transmission rate is greater than the second transmission rate; Determine the current screen refresh rate and color depth; A display resolution corresponding to the gaze area is determined according to the first transmission rate, the current screen refresh rate, and the color depth, and an actual resolution corresponding to the non-gaze area is determined according to the second transmission rate, the current screen refresh rate, and the color depth.

3. The image display method according to claim 2, wherein: The step of determining the display resolution corresponding to the gaze area according to the first transmission rate, the current screen refresh rate, and the color depth, and determining the actual resolution corresponding to the non-gaze area according to the second transmission rate, the current screen refresh rate, and the color depth comprises: Determine a first resolution according to a ratio of the first transmission rate to the current screen refresh rate, determine a second resolution according to a ratio of the first resolution to the color depth, and perform a square root operation on the second resolution to obtain a display resolution corresponding to the gaze area; and, determining a third resolution according to a ratio of the second transmission rate to the current screen refresh rate, determining a fourth resolution according to a ratio of the third resolution to the color depth, performing a square root operation on the fourth resolution, and obtaining an actual resolution corresponding to the non-attention area.

4. The image display method according to claim 2, wherein: The step of determining a first transmission rate of the attention area and a second transmission rate of the non-attention area comprises: Get the maximum transmission rate of the current system; Determine the transmission rate ratios corresponding to the attention area and the non-attention area; A first transmission rate corresponding to the attention area and a second transmission rate corresponding to the non-attention area are determined according to the maximum transmission rate and the transmission rate ratio.

5. The image display method according to claim 1, wherein: The step of determining the initial display resolution corresponding to the non-attention area according to the preset resolution of the display screen and the display resolution corresponding to the attention area comprises: A difference between a preset resolution of the display screen and a display resolution corresponding to the gaze area is determined as an initial display resolution corresponding to the non-gaze area.

6. The image display method according to claim 1, wherein: The step of determining the pixel magnification ratio corresponding to the non-attention area according to the actual resolution corresponding to the non-attention area and the initial display resolution comprises: The ratio of the actual resolution corresponding to the non-attention area to the initial display resolution is determined as the pixel magnification ratio corresponding to the non-attention area.

7. The image display method according to claim 1, wherein: Determining the fixation area in the image to be displayed includes: Determine the pupil position at the time when the picture to be displayed is displayed; A gaze area in the picture to be displayed is determined according to the pupil position.

8. The image display method according to claim 7, characterized in that: The step of determining the pupil position at the time of displaying the picture to be displayed comprises: determining the difference in relative position between the historical pupil position and the initial pupil position; Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions; The pupil position is determined according to the relative offset and the offset direction.

9. A head mounted display device, characterized in that: The head mounted display device comprises: a memory, a processor, and an image display program stored in the memory and executable on the processor, wherein when the image display program is executed by the processor, the steps of the image display method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image display program, and when the image display program is executed by a processor, the steps of the image display method according to any one of claims 1 to 8 are implemented.

Images