Control method of head-mounted display device, head-mounted display device and storage medium
By dividing the field of view according to the pupil position in the head-mounted display device and adjusting the pixel size, the edge connection problem in local high-definition display is solved, which improves the display effect and user experience, and reduces the device power consumption.
Patent Information
- Application Number
- CN202310661518.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
In the local high-definition display solution, when combining a normal background display screen and a high-definition focus display screen through an optical synthesizer, edge connection problems will occur, resulting in poor local high-definition display effect and poor user experience.
By determining the pupil position at the moment when the screen to be displayed is displayed, the first field of view that requires high-definition display and the second field of view that does not require high-definition display are divided, and the current pixel size of the second field of view is adjusted to the target pixel size , make it larger than the current pixel size, and control the head-mounted display device to display the screen to be displayed after adjusting the pixel size.
It realizes a local high-definition display effect without edge connection on the same screen, improving the display effect and reducing device power consumption.
Smart Images

Figure CN119065133A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of head-mounted display devices, and in particular to a control method for a head-mounted display device, a head-mounted display device and a storage medium. Background Art
[0002] In the partial HD display solution, an ordinary background display and a high-definition focusing display are combined through an optical synthesizer to achieve partial HD display. The clarity of the background display is lower than that of the high-definition focusing display. The background display deletes the display content of the gaze area according to the eye movement data, and the focusing display displays the deleted display content of the gaze area. However, when the display content is merged, there will be edge connection problems, resulting in poor partial HD display effect and poor user experience.
[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of the present invention is to provide a control method of a head-mounted display device, a head-mounted display device and a storage medium, aiming to achieve an improved display effect of a local high-definition display.
[0005] To achieve the above object, the present invention provides a method for controlling a head mounted display device, the method comprising:
[0006] Determine the pupil position at the time when the picture to be displayed is displayed;
[0007] Determine a first viewing area in the picture to be displayed according to the pupil position;
[0008] Adjusting a current pixel size of a second field of view area in the image to be displayed to a target pixel size according to the first field of view area, wherein the second field of view area is an area of the image to be displayed excluding the first field of view area, and the target pixel size is larger than the current pixel size;
[0009] Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.
[0010] Optionally, the step of determining the first viewing area in the picture to be displayed according to the pupil position includes:
[0011] Determining the human eye field of view at the display time according to the pupil position;
[0012] Determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position;
[0013] The first field of view area is determined according to the enlarged range and the field of view area of the human eye.
[0014] Optionally, before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the step further includes:
[0015] Obtaining the accuracy range of the pupil capture device for collecting pupil positions;
[0016] The capture error is determined according to the accuracy range.
[0017] Optionally, before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the step further includes:
[0018] Obtaining first position information when the human eye simulation camera simulates the movement of the human eye;
[0019] Acquire second position information of the human eye simulation camera collected by the pupil capture device when the human eye simulation camera simulates the movement of the human eye;
[0020] The capture error is determined according to the first position information and the second position information.
[0021] Optionally, the step of determining the pupil position at the time when the picture to be displayed is displayed includes:
[0022] determining the difference in relative position between the historical pupil position and the initial pupil position;
[0023] Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions;
[0024] The pupil position is determined according to the relative offset and the offset direction.
[0025] Optionally, before the step of adjusting the current pixel size of the second field of view in the image to be displayed to a target pixel size according to the first field of view, the method further includes:
[0026] Detecting the lumen value of the head mounted display device screen based on the light sensor of the head mounted display device;
[0027] The target pixel size is determined according to the lumen value.
[0028] Optionally, the step of determining the target pixel size according to the lumen value comprises:
[0029] When the lumen value is greater than or equal to a preset lumen value, determining that the current pixel size is increased by a first size;
[0030] When the lumen value is less than a preset lumen value, determining that the current pixel size is increased by a second size;
[0031] The target pixel size is determined according to the current pixel size and the first size, or according to the current pixel size and the second size, wherein the second size is larger than the first size.
[0032] Optionally, the head mounted device includes a micro control unit and a pupil capture device, and before the step of adjusting the current pixel size of the second field of view area in the to-be-displayed image to the target pixel size according to the first field of view area, the head mounted device includes a micro control unit and a pupil capture device, further comprising:
[0033] Receiving a video to be played sent by a video source, and determining the picture to be displayed according to the video to be played;
[0034] The step of determining the pupil position at the time of displaying the picture to be displayed comprises:
[0035] The micro-control unit receives the pupil movement position data sent by the pupil capture device, determines the pupil position according to the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device according to the pupil position;
[0036] The step of determining the first viewing area in the picture to be displayed according to the pupil position comprises:
[0037] Receiving the local high-definition display area sent by the micro-control unit;
[0038] Determine the first viewing field area according to the local high-definition display area;
[0039] The step of controlling the head mounted display device to display the picture to be displayed after the pixel size is adjusted comprises:
[0040] The picture to be displayed after the pixel size is adjusted is sent to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.
[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a head-mounted display device, which includes a memory, a processor, and a control program of the head-mounted display device stored in the memory and executable on the processor. When the control program of the head-mounted display device is executed by the processor, the steps of the control method of the head-mounted display device as described above are implemented.
[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a control program of a head-mounted display device is stored. When the control program of the head-mounted display device is executed by a processor, the steps of the control method of the head-mounted display device as described above are implemented.
[0043] The embodiment of the present invention proposes a control method for a head-mounted display device, a head-mounted display device and a storage medium, which determine the pupil position at the time of displaying a picture to be displayed; determine the first field of view area in the picture to be displayed according to the pupil position; adjust the current pixel size of the second field of view area in the picture to be displayed to the target pixel size according to the first field of view area, wherein the second field of view area is the area of the picture to be displayed except the first field of view area, and the target pixel size is larger than the current pixel size; and control the head-mounted display device to display the picture to be displayed after the pixel size is adjusted. Based on the pupil position at the time of displaying the picture to be displayed, the method divides the first field of view area that needs high-definition display and the second field of view area that does not need high-definition display in the same picture to be displayed, and adjusts the current pixel size of the second field of view area to the target pixel size with a larger pixel size, so that the clarity of the picture displayed in the second field of view area is lower than the clarity of the picture displayed in the first field of view area, and the two field of view areas can be displayed on the same screen without edge connection problems, so that the display effect of the local high-definition display of the head-mounted display device is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention;
[0045] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for controlling a head mounted display device of the present invention;
[0046] Figure 3 A schematic flow chart of another embodiment of a method for controlling a head mounted display device of the present invention;
[0047] Figure 4 This is a schematic diagram of a first application scenario involved in an embodiment of the present invention;
[0048] Figure 5 This is a schematic diagram of a second application scenario involved in an embodiment of the present invention;
[0049] Figure 6 This is a schematic diagram of a second application scenario involved in an embodiment of the present invention;
[0050] Figure 7 The figure is a schematic diagram of the structure of a head mounted display device according to an embodiment of the present invention.
[0051] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0053] In the related art, a solution of realizing partial high-definition display by combining a common background display screen and a high-definition focusing display screen through an optical synthesizer will cause edge connection problems when display contents are merged, thus resulting in poor display effect of partial high-definition display.
[0054] In order to achieve a local high-definition display effect, an embodiment of the present invention provides a control method for a head-mounted display device, a head-mounted display device, and a storage medium, wherein the main steps of the method include:
[0055] Determine the pupil position at the time when the picture to be displayed is displayed;
[0056] Determine a first viewing area in the picture to be displayed according to the pupil position;
[0057] Adjusting a current pixel size of a second field of view area in the image to be displayed to a target pixel size according to the first field of view area, wherein the second field of view area is an area of the image to be displayed excluding the first field of view area, and the target pixel size is larger than the current pixel size;
[0058] Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.
[0059] In this way, based on the pupil position, a first field of view area that needs high-definition display and a second field of view area that does not need high-definition display are divided in the same picture to be displayed, and the current pixel size of the second field of view area is adjusted to a target pixel size with a larger pixel size, so that the clarity in the second field of view area is lower than that in the first field of view area. While achieving local high-definition display, the two fields of view are not displayed separately, and there is no problem of edge connection of the displayed content, which improves the display effect of local high-definition display of the head-mounted display device.
[0060] The content to be protected by the claims of the present invention will be described in detail below with reference to the accompanying drawings.
[0061] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.
[0062] The terminal in the embodiment of the present invention may be a head mounted display device.
[0063] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The memory 1003 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1003 may also be a storage device independent of the aforementioned processor 1001.
[0064] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0065] like Figure 1 As shown, the memory 1003 as a computer storage medium may include an operating system and a control program of the head mounted display device.
[0066] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the control program of the head mounted display device stored in the memory 1003, and perform the following operations:
[0067] Determine the pupil position at the time when the picture to be displayed is displayed;
[0068] Determine a first viewing area in the picture to be displayed according to the pupil position;
[0069] Adjusting a current pixel size of a second field of view area in the image to be displayed to a target pixel size according to the first field of view area, wherein the second field of view area is an area of the image to be displayed excluding the first field of view area, and the target pixel size is larger than the current pixel size;
[0070] Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.
[0071] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0072] Determining the human eye field of view at the display time according to the pupil position;
[0073] Determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position;
[0074] The first field of view area is determined according to the enlarged range and the field of view area of the human eye.
[0075] Optionally, before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the step further includes:
[0076] Obtaining the accuracy range of the pupil capture device for collecting pupil positions;
[0077] The capture error is determined according to the accuracy range.
[0078] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0079] Obtaining first position information when the human eye simulation camera simulates the movement of the human eye;
[0080] Acquire second position information of the human eye simulation camera collected by the pupil capture device when the human eye simulation camera simulates the movement of the human eye;
[0081] The capture error is determined according to the first position information and the second position information.
[0082] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0083] determining the difference in relative position between the historical pupil position and the initial pupil position;
[0084] Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions;
[0085] The pupil position is determined according to the relative offset and the offset direction.
[0086] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0087] Detecting the lumen value of the head mounted display device screen based on the light sensor of the head mounted display device;
[0088] The target pixel size is determined according to the lumen value.
[0089] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0090] When the lumen value is greater than or equal to a preset lumen value, determining that the current pixel size is increased by a first size;
[0091] When the lumen value is less than a preset lumen value, determining that the current pixel size is increased by a second size;
[0092] The target pixel size is determined according to the current pixel size and the first size, or according to the current pixel size and the second size, wherein the second size is larger than the first size.
[0093] Further, the processor 1001 may call a control program of the head mounted display device stored in the memory 1003, and further perform the following operations:
[0094] Receiving a video to be played sent by a video source, and determining the picture to be displayed according to the video to be played;
[0095] The step of determining the pupil position at the time of displaying the picture to be displayed comprises:
[0096] The micro-control unit receives the pupil movement position data sent by the pupil capture device, determines the pupil position according to the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device according to the pupil position;
[0097] The step of determining the first viewing area in the picture to be displayed according to the pupil position comprises:
[0098] Receiving the local high-definition display area sent by the micro-control unit;
[0099] Determine the first viewing field area according to the local high-definition display area;
[0100] The step of controlling the head mounted display device to display the picture to be displayed after the pixel size is adjusted comprises:
[0101] The picture to be displayed after the pixel size is adjusted is sent to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.
[0102] The following is an explanation of the content claimed in the claims of the present invention through specific exemplary schemes, so that those skilled in the art can better understand the scope of protection of the claims of the present invention. It is understood that the following exemplary schemes do not limit the scope of protection of the present invention, but are only used to explain the present invention.
[0103] For example, refer to Figure 2 In one embodiment of the control method of the head mounted display device of the present invention, the control method of the head mounted display device comprises the following steps:
[0104] Step S10, determining the pupil position at the time of displaying the picture to be displayed;
[0105] In this embodiment, the head-mounted display device refers to a display device worn on the user's head, such as a virtual display device, an enhanced display device, etc. The user watches the content displayed on the screen of the head-mounted display device at a close distance. Due to the close distance, the field of view that the user can see is small and may not completely include the screen of the head-mounted display device. Therefore, the local high-definition display can reduce the resolution of the display content that the user cannot see or cannot see clearly, and retain the display effect of the display content that the user can see or focus on. In this way, the power consumption of the product can be reduced without changing the viewing effect of the user.
[0106] To distinguish between areas on the screen that need to reduce definition and areas that do not need to reduce definition, it is necessary to determine based on the user's pupil position, and it is necessary to determine the pupil position at the time when the picture to be displayed is displayed. Among them, the pupil position is the position of the current user's pupil of the head-mounted display device, and the head-mounted display device can detect the user's real-time pupil position, and then determine the pupil position at the time when the picture to be displayed is displayed. The pupil position at the time when the picture to be displayed is displayed can be the pupil position at the display time, and the pupil position at the adjacent time of the display time can also be used as the pupil position at the display time, wherein the interval time between the adjacent time and the display time is less than the preset time. Since the user's pupil position will not change too much in a short time, the pupil position at the adjacent time can be used as the pupil position at the display time. For example, if the interval time between the current time and the display time of the picture to be displayed is less than the preset time, the pupil position detected at the current time can be used as the pupil position at the time when the picture to be displayed is displayed.
[0107] Step S20, determining a first viewing area in the image to be displayed according to the pupil position;
[0108] In this embodiment, the time when the image to be displayed is displayed can be determined according to the pupil position. The vertical line between the center of the user's pupil and the display screen of the head-mounted display device can be used to construct the user's visual cone according to the vertical line and the human eye field angle. Figure 4 , generally, the best viewing angle of the human eye is 60 degrees. The viewing cone is constructed with 60 degrees as the angle of the cross section of the viewing cone, and the median line of the angle as the vertical line. According to the pupil position, the overlapping area between the corresponding viewing cone and the surface where the screen of the head-mounted display device is located, the local high-definition display area on the screen can be determined. The local high-definition display area is also the display area on the screen that the user can see or see clearly. The display content in the local high-definition display area can retain the best display effect to improve the user's viewing effect. Specifically, the surface where the screen is located can be determined according to the distance between the human eye and the screen, and the local high-definition display area of the screen can be determined according to the pupil position at the time of display of the picture to be displayed, the distance between the human eye and the screen, and the best viewing angle of the human eye.
[0109] Determine the display content of the picture to be displayed in the local high-definition display area, and the area in the picture to be displayed corresponding to the display content is the first field of view. First, determine the mapping relationship between the picture to be displayed and the display area of the screen, and according to the mapping relationship and the position of the local high-definition display area of the screen in the screen, the first field of view area in the picture to be displayed that will be displayed in the local high-definition display area can be determined.
[0110] Step S30, adjusting the current pixel size of the second field of view area in the image to be displayed to a target pixel size according to the first field of view area, wherein the second field of view area is an area of the image to be displayed excluding the first field of view area, and the target pixel size is larger than the current pixel size;
[0111] In this embodiment, the first field of view is the area that the user can see or see clearly when the picture to be displayed is displayed, and the area other than the first field of view in the picture to be displayed is the area that the user cannot see or cannot see clearly, that is, the second field of view. The picture in the second field of view of the picture to be displayed is determined according to the first field of view, the picture to be displayed is composed of an arrangement of pixels, and the pixel size of the pixels in the picture to be displayed is the current pixel size. The current pixel size is adjusted to the target pixel size, and the target pixel size is larger than the current pixel size. After adjusting to the target pixel size, the number of pixels per unit area in the second field of view becomes smaller, so that the clarity in the second field of view is reduced, and the corresponding power consumption during display is reduced.
[0112] Optionally, the step of adjusting the current pixel size of the second field of view area in the image to be displayed to the target pixel size according to the first field of view area may include:
[0113] Determine the target area occupied by the target pixel points in the second field of view after adjusting to the target pixel size, merge the current pixel points of the second field of view in the target area into the target pixel points, and when merging, determine the target pixel value of the target pixel point according to the pixel value corresponding to the current pixel point of the second field of view in the target area. For example, the average of the pixel values corresponding to the current pixel points in the second field of view in the target area is used as the target pixel value, and the target picture corresponding to the second field of view is constructed according to the target pixel value and the target area.
[0114] Step S40: Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.
[0115] In this embodiment, after the pixel size of the second field of view is adjusted to the target pixel size, a target image of the second field of view is generated accordingly, and the target image obtained after the pixel size is adjusted and the image corresponding to the first field of view without adjusting the pixel size are combined into an image to be displayed after the pixel size is adjusted. Since the current pixel size of the second field of view is adjusted to a larger target pixel size, the number of pixels per unit area of the second field of view is reduced, so that the clarity in the second field of view is reduced, and the corresponding display power consumption is reduced, while the pixel size of the first field of view that can be viewed or seen clearly by the user at the display time is still the current pixel size, and the clarity has not changed, and the clarity will be higher than the clarity of the second field of view, so that the viewing effect of the user will not be reduced, thereby achieving partial high-definition display.
[0116] In the technical solution disclosed in this embodiment, the pupil position at the time of displaying the picture to be displayed is determined; the first field of view area in the picture to be displayed is determined according to the pupil position; the current pixel size of the second field of view area in the picture to be displayed is adjusted to the target pixel size according to the first field of view area, wherein the second field of view area is the area of the picture to be displayed other than the first field of view area, and the target pixel size is larger than the current pixel size; the head-mounted display device is controlled to display the picture to be displayed after the pixel size is adjusted. In this way, after the first field of view area that needs high-definition display and the second field of view area that does not need high-definition display are divided in the same picture to be displayed based on the pupil position, the current pixel size of the second field of view area is adjusted to the target pixel size with a larger pixel size, so that the clarity in the second field of view area is lower than that in the first field of view area, and the two field of view areas can be displayed on the same screen without edge connection problems, so that the display effect of the local high-definition display of the head-mounted display device is improved, and the effect of reducing the power consumption of the device can also be achieved by the local high-definition display.
[0117] Further, step S10 includes:
[0118] Determining the human eye field of view at the display time according to the pupil position;
[0119] Determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position;
[0120] The first field of view area is determined according to the enlarged range and the field of view area of the human eye.
[0121] There may be a certain capture error when capturing the pupil position, resulting in a position difference between the determined pupil position at the time of display of the image to be displayed and the actual pupil position, which leads to the fact that the human eye field of view determined according to the pupil position is not visible to the user or the actual human eye field of view. Directly using it as the first field of view may have a certain error, so the capture error needs to be taken into account in determining the first field of view according to the human eye field of view. First, the capture error of the pupil position is determined. The capture error is the difference between the determined pupil position and the actual pupil position, and the capture error can be a preset parameter. Since the relative direction between the pupil position and the actual pupil position cannot be determined, the distance between the boundary points in all directions of the human eye's field of view and its center must be enlarged, and the enlarged distance must be no less than the capture error. After the boundary points in each direction are enlarged, it is equivalent to expanding the range of the human eye's field of view with the center of the human eye's field of view as the midpoint. The enlarged range of this human eye's field of view is determined based on the capture error, and the human eye's field of view and its enlarged range are used as the area corresponding to the first field of view on the screen, that is, the local high-definition display area, and the first field of view is determined based on the area corresponding to the first field of view on the screen and the above-mentioned mapping relationship.
[0122] It should be noted that in the AA area of the head mounted display device, the distances of the boundary points in all directions of the human eye field of view relative to the center are not less than the capture error, but they may not be equal. For example, Figure 5 According to the human eye field of view angle and pupil position, the human eye field of view area that the user can see is determined as a circular area S, and the circular area S is expanded to L in the upper, lower, left and right directions, where L is a value greater than or equal to the capture error, to form an area Q corresponding to the first field of view area on the square screen.
[0123] This solves the problem that the area corresponding to the first field of view on the screen cannot be accurately determined due to pupil error, resulting in the clarity of the display image in the area that the user can see or see clearly being erroneously reduced, thereby improving the user's viewing experience and thus improving the display effect of local high-definition display.
[0124] Furthermore, before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the method further includes:
[0125] Obtaining the accuracy range of the pupil capture device for collecting pupil positions;
[0126] The capture error is determined according to the accuracy range.
[0127] The pupil position is collected by a pupil capture device. The pupil capture device can capture the pupil position at a certain moment, and can also track the pupil by determining the pupil position at multiple consecutive moments to obtain pupil movement position data. For example, a shooting device can be used as a pupil capture device, wherein the shooting device can capture the image of the human eye, and the pupil can be captured based on the human eye image to determine the pupil position. The pupil capture device has a certain accuracy range, and the accuracy of the pupil capture device determines the key to the position difference between the pupil position and the actual pupil position. The pupil capture device is generally associated with its accuracy range and saved. The accuracy range of the pupil capture device can be obtained, and the capture error can be determined based on the accuracy range. For example, if the accuracy range corresponding to the pupil capture device is + / -0.05mm, the capture error can be greater than 0.05mm. In this way, the capture error can be quickly determined through the accuracy range of the pupil capture device, and the efficiency of determining the capture error is improved, thereby improving the efficiency of local high-definition display and reducing costs.
[0128] Furthermore, before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the method further includes:
[0129] Obtaining first position information when the human eye simulation camera simulates the movement of the human eye;
[0130] Acquire second position information of the human eye simulation camera collected by the pupil capture device when the human eye simulation camera simulates the movement of the human eye;
[0131] The capture error is determined according to the first position information and the second position information.
[0132] The capture error can also be determined through production testing, and the human eye simulation camera is used to simulate the movement of the human eye. When the human eye simulation camera is simulating the human eye, the human eye simulation camera can move according to the image it captures and the preset first position information, obtain the preset first position information or other methods, and determine its own first position information in motion. When the human eye simulation camera is in the first position information, the pupil capture device simultaneously collects the second coordinate position of the human eye simulation camera, that is, the time when the first position information appears is the same as the time when the second position information is collected. The first position information can be used as the actual pupil position, and the second position information can be used as the pupil position captured by the pupil capture device. The relative distance between the position determined by the human eye simulation camera and the position of the human eye simulation camera captured by the pupil capture device can be determined based on the first position information and the second position information, and the capture error is determined based on the relative distance of the positions.
[0133] In order to improve the accuracy of determining the capture error, the first position information may include the position determined by the human eye simulation camera at multiple moments, and the position of the human eye simulation camera is also collected by the pupil capture device at the corresponding moment, so the second position information includes the position of the human eye simulation camera collected by the pupil capture device at multiple moments, and the difference analysis is performed accordingly to obtain multiple relative distances, and the relative distances are analyzed for normal distribution to determine the capture error. For example, a human eye simulation camera with a viewing angle of 60° (the angle can be adjusted randomly, including the maximum boundary position to which the eyeball can move) is installed at the position of the dummy eye to simulate the movement of the human eye, and the position coordinate A of the human eye simulation camera is recorded, and at the same time, the eyeball capture camera (pupil capture device) captures the coordinate position B of the human eye simulation camera, and the recorded position coordinate A of the human eye simulation camera and the coordinate position B of the human eye simulation camera captured by the eyeball capture camera are calculated for difference, and the same test action is repeated to measure 100 sets of data, and the difference calculation results of the 100 sets of data are analyzed for normal distribution diagrams to find out, find out the reasonable value C of the difference calculation result, and determine that the capture error is greater than C.
[0134] This testing method accurately determines the capture error, improves the accuracy of determining the capture error, and thus improves the accuracy of local high-definition display.
[0135] Furthermore, the step S10 includes:
[0136] determining the difference in relative position between the historical pupil position and the initial pupil position;
[0137] Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions;
[0138] The pupil position is determined according to the relative offset and the offset direction.
[0139] In this embodiment, the pupil position at the moment when the picture to be displayed is displayed may be the pupil position at the moment when the picture to be displayed is displayed. Since the display moment has not occurred, the pupil position at the moment of display may be predicted based on the historical pupil position. First, the initial pupil position is determined. The initial pupil position may be the pupil position of the human eye looking straight ahead, that is, the center position of the eyeball, or the pupil position collected before the historical pupil position is collected. The historical pupil position and the actually collected pupil position may be one or more pupil positions collected the latest including the current pupil position, including the pupil position collected at the current moment. The difference between the relative position of the historical pupil position and the initial position is calculated, thereby calculating the relative offset and offset direction of the pupil relative to the initial position. The movement change of the pupil may be fitted based on the relative offset and offset direction, and the pupil position at the moment of display may be determined based on the movement change.
[0140] For a better understanding, a specific application scenario is provided:
[0141] When the user wears the VR headset, the eye tracking camera is turned on immediately to capture the left and right pupil distances and automatically adjust the left and right display centers of the headset. After the adjustment is complete, the eye center position (X0, Y0) is recorded as the initial pupil position and recorded in the memory. The movement of the pupil is tracked, and the human eye image is captured at a certain frequency and transmitted to the memory. The current pupil position is compared with the current pupil position. Figure 6 The difference between (X1, Y1)(X2, Y2)(X3, Y3)...(Xn, Yn) and the initial position (X0, Y0) is used to calculate the relative offset and offset direction of the pupil, and determine the pupil position at the moment when the image to be displayed is displayed.
[0142] In this way, the pupil position at the moment of displaying the image to be displayed is predicted through the difference between the relative positions of the historical pupil position and the initial pupil position, thereby improving the accuracy of determining the first field of view and improving the viewing effect of the local high-definition display.
[0143] Optionally, in order to save power consumption, it is also possible to determine whether the pupil position at the time the picture to be displayed is displayed will change significantly, thereby determining whether the first field of view will change significantly. When a significant change occurs, the pupil position at the time the picture to be displayed is determined, and then the first field of view is determined. When no significant change is sent, there is no need to determine the first field of view based on the pupil position of the picture to be displayed, but the first field of view of the currently displayed picture is used as the first field of view of the picture to be displayed.
[0144] Specifically, after step S10, the pupil position offset of the pupil position at the moment of display of the image to be displayed relative to the historical pupil position is determined. When the pupil position offset is greater than the first preset offset, step S20 is executed; otherwise, the first field of view area determined at the most recent moment is used as the first field of view area of the image to be displayed, and step S30 is executed.
[0145] Or before step S10, determine the historical image key points of the historical display screen, and determine the image key points of the screen to be displayed. The image key points can be the focus of the image or the focus of the image content. The user's gaze point is likely to fall on the image key points, and the user's gaze point is related to the pupil position. For example, when a face appears in the image, the user's pupil will be oriented toward the face position. If the historical display screen and the screen to be displayed are from the same video, then the historical image key points of the historical display screen and the image key points of the screen to be displayed can be determined based on the moving objects in the video. When the offset of the image key points of the screen to be displayed relative to the historical image key points is greater than the second preset offset, it indicates that the pupil position will change significantly when the screen to be displayed is displayed relative to the pupil position at the time when the historical display image is displayed, and step S10 can be executed, otherwise the first field of view area determined at the most recent moment is used as the first field of view area of the screen to be displayed.
[0146] Optionally, refer to Figure 3 Based on any of the above embodiments, in another embodiment of the control method of the head mounted display device of the present invention, before step S30, the control method of the head mounted display device further includes:
[0147] Step S50: detecting the lumen value of the head mounted display device screen based on the light sensor of the head mounted display device;
[0148] In this embodiment, the target pixel size is determined according to the usage scenario or display scenario of the head mounted display device, and the usage scenario or display scenario can be determined according to the image to be displayed, or according to the lumen value of the head mounted display device screen when displaying. A light sensor can be set in the built-in head mounted display device, and the lumen value of the head mounted display device screen can be measured in real time according to the light sensor, which is also the lumen value of the corresponding display scenario.
[0149] Step S60: Determine the target pixel size according to the lumen value.
[0150] In this embodiment, the lumen value of the head mounted display device screen is detected by a light sensor. When the lumen value is high, the display resolution requirement is reduced, and the second field of view area has a larger space for clarity reduction, so the target pixel size can be larger; when the lumen value is low, the display resolution requirement is increased, and the second field of view area has a smaller space for clarity reduction, so the target pixel size can be smaller than the target pixel size when the lumen value is higher. The target pixel size can be determined based on the preset relationship between the lumen value and the target size, and the currently detected lumen value.
[0151] Optionally, the preset relationship between the lumen value and the target pixel size may be a negatively correlated growth relationship, where the target pixel size decreases as the lumen value increases and increases as the lumen value decreases, thereby more accurately matching the target pixel size of the display scene.
[0152] In the scheme disclosed in this embodiment, the brightness of the display scene or usage scene of the head-mounted display device is determined according to the detected screen lumen value to quickly and accurately determine the resolution requirement, and then the target pixel size is determined according to the lumen value, so that the target pixel size meets the user's requirements for clarity and further improves the display effect of the local high-definition display.
[0153] Further, the step of determining the target pixel size according to the lumen value includes:
[0154] When the lumen value is greater than or equal to a preset lumen value, determining that the current pixel size is increased by a first size;
[0155] When the lumen value is less than a preset lumen value, determining that the current pixel size is increased by a second size;
[0156] The target pixel size is determined according to the current pixel size and the first size, or according to the current pixel size and the second size, wherein the second size is larger than the first size.
[0157] The lumen value can also be compared with the preset lumen value. When the lumen value is greater than or equal to the preset lumen value, the resolution requirement will also increase, and it is determined that the current pixel size is increased by a first size. When the lumen value is less than the preset lumen value, the resolution requirement will also decrease, and it is determined that the current pixel size is increased by a second size, and the second size is greater than the first size, so that when the lumen value is greater than or equal to the preset lumen value, the corresponding target pixel size is smaller than the corresponding target pixel size when the lumen value is less than the preset lumen value.
[0158] In this way, by comparing the lumen value with the preset lumen value, the target pixel size can be quickly determined, the adjustment speed of the second field of view area can be increased, and the efficiency of local high-definition display can be improved.
[0159] Optionally, based on any of the above embodiments, in another embodiment of the control method of a head mounted display device of the present invention, before step S10, the control method of the head mounted display device further includes:
[0160] Receiving a video to be played sent by a video source, and determining the picture to be displayed according to the video to be played;
[0161] Step S10 includes:
[0162] The micro-control unit receives the pupil movement position data sent by the pupil capture device, determines the pupil position according to the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device according to the pupil position;
[0163] Step S20 includes:
[0164] Receiving the local high-definition display area sent by the micro-control unit;
[0165] Determine the first viewing field area according to the local high-definition display area;
[0166] Step S40 includes:
[0167] The picture to be displayed after the pixel size is adjusted is sent to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.
[0168] Reference Figure 7 The head-mounted display device may include a screen (the screen may include a screen divided into two display areas corresponding to the left and right eyes, or may include two screens corresponding to the left and right eyes respectively, and processed with the display process corresponding to a single eye), a software processing system, a micro-control unit connected between the software processing system and the screen, and a pupil capture device connected to the micro-control unit and used to collect the pupil positions of the left and right eyes.
[0169] The control method provided in this embodiment can be applied to a software processing system, which receives a high-definition video to be played sent by a video source, and determines multiple image frames of the video to be played as multiple pictures to be displayed according to the video to be played; the pupil capture device can collect pupil positions at multiple moments, and send the pupil positions at multiple moments to the micro-control unit or send the pupil positions to the micro-control unit every time a pupil position is collected at a moment, and one or more pupil positions sent to the micro-control unit constitute pupil movement position data, and the micro-control unit receives the pupil movement position data collected by the pupil capture device; the micro-control unit determines the display time of the picture to be displayed according to the pupil movement position data The pupil position is determined, and then the micro-control unit determines the display time of the picture to be displayed according to the pupil position at the display time of the picture to be displayed, and the local high-definition display area on the screen, and the micro-control unit sends the local high-definition display area to the software processing system; the software processing system determines the first field of view area in the picture to be displayed according to the local high-definition display area, and adjusts the current pixel size corresponding to the second field of view area of the picture to be displayed except the first field of view area to the target pixel size according to the first field of view area in the picture to be displayed, to obtain the picture to be displayed after the pixel size is adjusted, and the software processing system sends the picture to be displayed to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.
[0170] Optionally, the head-mounted display device may include a light sensor, with the probe side of the light sensor facing the screen, collecting the lumen value of the screen, and sending the lumen value to the software processing system. The software processing system determines the target pixel size corresponding to the second field of view area through the lumen value, so that the clarity of the adjusted second field of view area is more in line with viewing requirements, thereby improving the display effect of the local high-definition display.
[0171] This way, the display tasks and the display screen processing processes are separated, which can reduce the loss of CPU and GPU computing power on the screen display part, reduce the power consumption of the screen display part, and allow the computing power and power consumption of the head-mounted display device to be used more effectively, thereby improving the effect of local high-definition display.
[0172] In addition, an embodiment of the present invention further proposes a head-mounted display device, which includes a memory, a processor, and a control program of the head-mounted display device stored in the memory and executable on the processor. When the control program of the head-mounted display device is executed by the processor, the steps of the control method of the head-mounted display device described in the above embodiments are implemented.
[0173] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a control program for a head-mounted display device is stored. When the control program for the head-mounted display device is executed by a processor, the steps of the control method for the head-mounted display device described in the above embodiments are implemented.
[0174] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0175] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, optical disk) as described above, and includes a number of instructions for enabling the head-mounted display device to execute the methods described in each embodiment of the present invention.
[0177] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for controlling a head mounted display device, It is characterized in that The control method of the head mounted display device comprises: Determine the pupil position at the time when the picture to be displayed is displayed; Determine a first viewing area in the picture to be displayed according to the pupil position; Adjusting a current pixel size of a second field of view area in the image to be displayed to a target pixel size according to the first field of view area, wherein the second field of view area is an area of the image to be displayed excluding the first field of view area, and the target pixel size is larger than the current pixel size; Control the head mounted display device to display the image to be displayed after the pixel size is adjusted.
2. The method for controlling a head mounted display device according to claim 1, It is characterized in that The step of determining the first viewing area in the picture to be displayed according to the pupil position comprises: Determining the human eye field of view at the display time according to the pupil position; Determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position; The first field of view area is determined according to the enlarged range and the field of view area of the human eye.
3. The method for controlling a head mounted display device according to claim 2, It is characterized in that Before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the method further includes: Obtaining the accuracy range of the pupil capture device for collecting pupil positions; The capture error is determined according to the accuracy range.
4. The method for controlling a head mounted display device according to claim 2, It is characterized in that Before the step of determining the enlarged range of the human eye field of view according to the capture error corresponding to the pupil position, the method further includes: Obtaining first position information when the human eye simulation camera simulates the movement of the human eye; Acquire second position information of the human eye simulation camera collected by the pupil capture device when the human eye simulation camera simulates the movement of the human eye; The capture error is determined according to the first position information and the second position information.
5. The method for controlling a head mounted display device according to claim 1, It is characterized in that The step of determining the pupil position at the time of displaying the picture to be displayed comprises: determining the difference in relative position between the historical pupil position and the initial pupil position; Determining a relative offset and an offset direction of the pupil according to the difference in the relative positions; The pupil position is determined according to the relative offset and the offset direction.
6. The method for controlling a head mounted display device according to claim 1, It is characterized in that Before the step of adjusting the current pixel size of the second field of view in the image to be displayed to the target pixel size according to the first field of view, the method further includes: Detecting the lumen value of the head mounted display device screen based on the light sensor of the head mounted display device; The target pixel size is determined according to the lumen value.
7. The method for controlling a head mounted display device according to claim 6, It is characterized in that The step of determining the target pixel size according to the lumen value comprises: When the lumen value is greater than or equal to a preset lumen value, determining that the current pixel size is increased by a first size; When the lumen value is less than a preset lumen value, determining that the current pixel size is increased by a second size; The target pixel size is determined according to the current pixel size and the first size, or according to the current pixel size and the second size, wherein the second size is larger than the first size.
8. The method for controlling a head mounted display device according to claim 1, It is characterized in that The head mounted device comprises a micro control unit and a pupil capture device, and before the step of adjusting the current pixel size of the second field of view in the to-be-displayed image to the target pixel size according to the first field of view, the step further comprises: Receiving a video to be played sent by a video source, and determining the picture to be displayed according to the video to be played; The step of determining the pupil position at the time of displaying the picture to be displayed comprises: The micro-control unit receives the pupil movement position data sent by the pupil capture device, determines the pupil position according to the pupil movement position data, and determines a local high-definition display area in the screen of the head-mounted display device according to the pupil position; The step of determining the first viewing area in the picture to be displayed according to the pupil position comprises: Receiving the local high-definition display area sent by the micro-control unit; Determine the first viewing field area according to the local high-definition display area; The step of controlling the head mounted display device to display the picture to be displayed after the pixel size is adjusted comprises: The picture to be displayed after the pixel size is adjusted is sent to the micro-control unit, and the micro-control unit controls the screen to display the picture to be displayed after the pixel size is adjusted.
9. A head mounted display device, It is characterized in that The head-mounted display device includes: a memory, a processor, and a control program of the head-mounted display device stored in the memory and executable on the processor. When the control program of the head-mounted display device is executed by the processor, the steps of the method for controlling the head-mounted display device according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a control program for a head-mounted display device, and when the control program for the head-mounted display device is executed by a processor, the steps of the method for controlling a head-mounted display device according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Image processing apparatus and method
CN105631926A
Using pupil location to correct optical lens distortion
CN109891296A
Image storage method and device, electronic equipment and storage medium
CN112911186A
Control method and device, mobile terminal and storage medium
CN113129801A
Display control method of virtual reality equipment, virtual reality equipment and storage medium
CN114911445A
