Screen position correction method, device and head mounted display device

Abstract

The present invention discloses a method, device and head-mounted display device for correcting the position of a picture; the method comprises: collecting eye data of a user; determining whether the user is in a non-viewing state according to the eye data; if it is determined that the user is in a non-viewing state, adjusting the position of the picture to an initial position. The present invention determines the viewing state of the user based on the eye data of the user collected in real time, and corrects the position of the picture when the user is in a non-viewing state. The overall process is simple, fast and imperceptible to the user, which can significantly improve the use effect and user comfort of the head-mounted display device.

Description

Technical Field

[0001] The present invention relates to the field of simulation display technology, and in particular to a picture position correction method, device and head-mounted display equipment. Background Art

[0002] With the development of virtual reality (VR) and augmented reality (AR) technologies, more and more head-mounted display devices using VR and AR technologies have entered people's daily lives. For existing head-mounted display devices, due to gyroscope accuracy and algorithm errors, most head-mounted display devices have image drift and twisting during use. After a period of use, the image will deviate from the initial position and need to be manually moved or the viewing angle manually reset to achieve image position correction, which seriously affects the sense of immersion and reduces the comfort of the experience. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a screen position correction method, device and head-mounted display device, which can realize simple, fast and user-imperceptible screen position correction, and significantly improve the use effect and user comfort of the head-mounted display device.

[0004] Based on the above purpose, the present invention provides a method for correcting the position of a picture, which is applied to a head mounted display device, comprising:

[0005] Collect user's eye data;

[0006] determining whether the user is in a non-viewing state according to the eye data;

[0007] If it is determined that the user is not in a viewing state, the screen position is adjusted to an initial position.

[0008] In some optional embodiments, the eye data is: electrical pulses of periocular muscles;

[0009] The determining, based on the eye data, whether the user is in a non-viewing state comprises:

[0010] Determining whether the user blinks according to the electrical pulses of the periocular muscles;

[0011] If it is determined that the user blinks, it is determined that the user is in a non-viewing state.

[0012] In some optional embodiments, the eye data is: an eye image;

[0013] The determining, based on the eye data, whether the user is in a non-viewing state comprises:

[0014] performing pupil location on the eye image;

[0015] If the pupil position cannot be located, it is determined that the user is in a non-viewing state.

[0016] In some optional embodiments, the eye data is: an eye image;

[0017] The determining, based on the eye data, whether the user is in a non-viewing state comprises:

[0018] Performing pupil positioning on the eye image to determine the pupil position;

[0019] The pupil position is compared with a preset standard position, and if the difference between the pupil position and the standard position exceeds a preset threshold, it is determined that the user is in a non-viewing state.

[0020] In some optional implementations, after adjusting the screen position to the initial position, the method further includes:

[0021] During the preset time interval, the user's eye data will no longer be collected.

[0022] On the other hand, the present invention also provides a screen position correction device, which is applied to a head mounted display device, comprising:

[0023] A collection component, used to collect eye data of the user;

[0024] The control component is used to determine whether the user is in a non-viewing state according to the eye data; if it is determined that the user is in a non-viewing state, the image position is adjusted to an initial position.

[0025] In some optional embodiments, the acquisition component is an eye movement sensor; the eye movement sensor is arranged at the part where the head-mounted display device contacts the user's eyes.

[0026] In some optional embodiments, the acquisition component is an infrared camera disposed inside the head-mounted display device.

[0027] In some optional embodiments, the apparatus further includes: at least one infrared lamp disposed inside the head mounted display device.

[0028] In another invention, the present invention also provides a head mounted display device, comprising: a screen position correction device as described in any one of the above items.

[0029] From the above description, it can be seen that the screen position correction method, device and head-mounted display device provided by the present invention judge the user's viewing state based on the user's eye data collected in real time, and correct the screen position when the user is in a non-viewing state. The overall process is simple, fast and user-imperceptible, which can significantly improve the use effect and user comfort of the head-mounted display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a flow chart of a method for correcting picture position according to an embodiment of the present invention;

[0032] Figure 2 for Figure 1 A flowchart of a specific embodiment of step 102;

[0033] Figure 3 for Figure 1 A flowchart of another specific embodiment of step 102;

[0034] Figure 4 for Figure 1 A flowchart of another specific embodiment of step 102;

[0035] Figure 5 is a flow chart of a method for correcting picture position according to another embodiment of the present invention;

[0036] Figure 6 It is a structural schematic diagram of a picture position correction device according to an embodiment of the present invention;

[0037] Figure 7 It is a schematic diagram of the overall structure of the picture position correction device according to an embodiment of the present invention when it is applied to a head mounted display device;

[0038] Figure 8 This is a schematic diagram of the overall structure of a picture position correction device according to another embodiment of the present invention when applied to a head mounted display device. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0040] An embodiment of the present invention provides a method for correcting the position of a picture, which is applied to a head-mounted display device to solve the problems of picture drift and twisting that occur during the use of an existing head-mounted display device. Specifically, to solve the problems of picture drift and twisting, it is necessary to correct the position of the picture, that is, to adjust the position of the picture back to a preset standard position; when the position of the picture is adjusted to the standard position, if the user is watching the picture, the picture will flicker and shake, affecting the viewing effect. The applicant has found that based on the application scenario when the user uses the head-mounted display device, the user's eye state can be determined to further determine the user's viewing state. Therefore, the inventive concept of the present invention is: based on the collection of the user's eye data, the user's eye state is known, and then the user's viewing state is determined, and when the user is in a non-viewing state, the picture position is corrected.

[0041] Based on the above invention concept, reference Figure 1 The picture position correction method of this embodiment includes the following steps:

[0042] Step 101: Collect user's eye data.

[0043] In this step, in order to know the user's eye status, the user's eye data is first collected. Specifically, the collection of eye data can be achieved in a variety of ways. For example, through the image method, that is, collecting eye images; the device for collecting eye images can be a camera built into the head-mounted display device, or an image acquisition device additionally installed on the head-mounted display device, which is specifically used to capture eye images. In addition, eye data can also be collected in the form of vital sign data, such as muscle electrical pulses; during collection, the user's eye muscle electrical pulses are collected through sensors set on the head-mounted display device.

[0044] Step 102: determining whether the user is in a non-viewing state according to the eye data;

[0045] Step 103: If it is determined that the user is not viewing the video, the screen position is adjusted to an initial position.

[0046] In the above steps, it is determined whether the user is in a non-viewing state based on the collected eye data. Based on the application scenario when the user uses the head-mounted display device, when the user looks at the screen, it can be determined as a viewing state. When the user is not looking at the screen, or blinks or closes his eyes, it can be determined as a non-viewing state. When it is determined that the user is in a viewing state, the screen position is not corrected to ensure the user's viewing effect. When it is determined that the user is in a non-viewing state, the screen position is adjusted to the initial position to complete the correction of the screen. At this time, the user's attention is not on the screen, so that the correction process of the screen is carried out without the user's awareness, thereby ensuring the user's viewing effect.

[0047] In an optional embodiment, the collected eye data is: electrical pulses of muscles around the eye. Figure 2 In the above-mentioned embodiment, step 102 specifically includes:

[0048] Step 201, judging whether the user blinks according to the electrical pulses of the periocular muscles;

[0049] Step 202: If it is determined that the user blinks, it is determined that the user is in a non-viewing state.

[0050] According to the application scenario of the method of this embodiment, when the user blinks, it can be determined that he is in a non-viewing state. According to medical statistics, the blinking action of humans is called a "blinking reflex", and the most common one is involuntary blinking. Under normal circumstances, involuntary blinking is completed unconsciously, usually once every 2-6 seconds, and each blink takes about 0.2-0.4 seconds. In addition, there is a period of about 0.2s after blinking, which is called visual inertia. During this period of time, the picture seen by the human eye is a residual impression of the picture before blinking, which is not very clear. In this embodiment, the above phenomenon is used to correct the picture position during the visual buffer time when the user blinks and after blinking. When correcting the picture position, the adjustment amount of adjusting the picture position to the initial position is generally about 1 / 10°. The fast and small picture position movement combined with the visual buffer time can make the user unaware of the picture position correction process, thereby achieving the technical effect of this embodiment.

[0051] In this embodiment, an eye movement sensor can be set on the head mounted display device to collect the user's eye muscle electrical pulses. The collection electrodes of the eye movement sensor can detect the tiny electrical pulses emitted by the muscles that control the eyes, and then use an algorithm to translate the electrical pulse signals into the movements of the human eye, thereby determining whether the user has blinked. Among them, when determining whether the user has blinked by using the eye movement sensor, the structure of the eye movement sensor and the algorithm involved are prior art and will not be described in detail in this application.

[0052] It should be noted that, based on the application scenario of the method of this embodiment, the user may also temporarily close his eyes to rest. At this time, it can be considered that the user has blinked for a long time. Therefore, it is obvious that the method of this embodiment can be used to determine whether the user has closed his eyes, and when it is determined that the user has closed his eyes, it is determined that the user is in a non-viewing state.

[0053] In an optional embodiment, the collected eye data is: eye image. Figure 3 In the above-mentioned embodiment, step 102 specifically includes:

[0054] Step 301, performing pupil location on the eye image;

[0055] Step 302: If the pupil position cannot be located, it is determined that the user is in a non-viewing state.

[0056] In this step, the user's eye image can be captured by a camera provided by the head-mounted display device or an image capture device installed on the head-mounted display device and specifically used to capture the user's eye image.

[0057] After the eye image is collected, the pupil is located for the eye image through image recognition and processing technology. When the pupil position cannot be located in the eye image, it indicates that there is no image corresponding to the pupil in the collected eye image, and it can be determined that the current user is blinking or closing the eyes, and accordingly it can be determined that the user is in a non-viewing state. Among them, the relevant algorithms involved in pupil location through image recognition and processing technology for eye images are prior arts and will not be described in detail in this application.

[0058] In an optional embodiment, the collected eye data is: eye image. Figure 4 In the above-mentioned embodiment, step 102 specifically includes:

[0059] Step 401, performing pupil location on the eye image to determine the pupil position;

[0060] Step 402: Compare the pupil position with a preset standard position. If the difference between the pupil position and the standard position exceeds a preset threshold, it is determined that the user is in a non-viewing state.

[0061] According to the application scenario of the method of this embodiment, during the viewing process, the user may not pay attention to the screen due to the playing content or other reasons. At this time, the user's line of sight will point to other directions outside the screen or to the edge of the screen. At this time, it can be determined that the user is in a non-viewing state. Correspondingly, in the viewing state, the user's line of sight will point to the middle of the screen.

[0062] In this embodiment, similar to the previous embodiment, after the eye image is collected, the pupil is located for the eye image through image recognition and processing technology, and the pupil position is determined. Based on the characteristics of the user's viewing state, the middle position of the picture is corresponded to a pre-set standard position. After the pupil position is determined, the pupil position is further compared with the standard position to obtain the difference between the two. In addition, in this embodiment, a preset threshold is also set; the preset threshold corresponds to the extreme range of the user's line of sight deviating from the middle position of the picture in actual application. When the difference between the pupil position and the standard position exceeds the preset threshold, it is considered that the user's attention is not on the picture, and then it is determined that the user is in a non-viewing state.

[0063] In an alternative embodiment, reference Figure 5 , after step 103 of the above embodiment, further comprising the steps of:

[0064] Step 104: Within a preset time interval, the user's eye data is no longer collected.

[0065] Considering that the drift and twist of the picture in the prior art are gradually accumulated after the head-mounted display device is used for a period of time. Therefore, after executing step 103 and completing the correction of the picture position, if the method of this embodiment is continuously and repeatedly executed, it may cause a waste of resources. Therefore, in this embodiment, after step 103 is executed, step 104 is continued to be executed. That is, within the preset time interval, the user's eye data is no longer collected, thereby temporarily stopping the execution of the method of this embodiment. Among them, the preset time interval can be obtained based on experience or based on the statistics of the user's actual usage data; for example, statistics are made on how long it takes for an average user to continuously use the head-mounted display device before the phenomenon of drift and twist of the picture occurs.

[0066] Based on the same inventive concept, the present invention also provides a picture position correction device. Figure 6 , the picture position correction device comprises:

[0067] The collection component 601 is used to collect the user's eye data;

[0068] The control component 602 is used to determine whether the user is in a non-viewing state based on the eye data; if it is determined that the user is in a non-viewing state, adjust the image position to an initial position.

[0069] In this embodiment, the acquisition component 601 may be an image acquisition device or a sensor.

[0070] In this embodiment, the control component 602 is an electronic component with data processing function to realize the above functions. Specifically, the control component 602 can be a separate chip to realize the above functions, which is arranged on the head-mounted display device and electrically connected to the existing control chip in the head-mounted display device. In addition, the above functions can also be software-implemented into the control chip of the head-mounted display device, that is, the control component 602 in this embodiment can also be the existing control chip in the head-mounted display device.

[0071] The apparatus of the above-mentioned embodiment is used to implement the corresponding method in the above-mentioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0072] In an alternative embodiment, reference Figure 7In the image position correction device, the acquisition component is an eye movement sensor 6011. The eye movement sensor is arranged at the part where the head mounted display device 001 contacts the user's eyes. Figure 7 In this embodiment, the part of the head mounted display device 001 that contacts the user's eyes is the face-attaching foam 0011, and the eye movement sensor 6011 is arranged on the face-attaching foam 0011, and corresponding to the user's eyes, two eye movement sensors 6011 are arranged. The eye movement sensor 6011 is connected to the control chip 0012 of the head mounted display device 001 through a flexible printed circuit (FPC).

[0073] In an alternative embodiment, reference Figure 8 In the picture position correction device, the acquisition component is an infrared camera 6012 arranged inside the head mounted display device 001. The infrared camera 6012 is used to acquire the user's eye image.

[0074] Furthermore, in this embodiment, the picture position correction device also includes: at least one infrared lamp 6013 disposed inside the head mounted display device 001. By using the infrared lamp 6013, infrared light is added to the space inside the head mounted display device 001, which can effectively improve the clarity of the eye image collected by the infrared camera 6012, thereby ensuring the effect of picture position correction.

[0075] Based on the same inventive concept, the present invention also provides a head mounted display device, which includes the screen position correction device as described in any one of the above embodiments.

[0076] Since the screen position correction device described in any of the above embodiments is applied, the head-mounted display device of this embodiment can obviously realize screen position correction simply, quickly and without the user noticing, which significantly improves the use effect and user comfort of the head-mounted display device.

[0077] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present invention, the above embodiments or technical features in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0078] In addition, to simplify the description and discussion, and in order not to obscure the present invention, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the devices may be shown in the form of block diagrams to avoid obscuring the present invention, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the present invention will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present invention, it will be apparent to those skilled in the art that the present invention may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0079] Although the invention has been described in conjunction with specific embodiments of the invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0080] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for correcting the position of a picture, applied to a head mounted display device, It is characterized in that include: Collect user's eye data; Determine whether the user is in a non-viewing state according to the eye data; wherein, determine whether the user blinks according to the electric pulses of the periocular muscles included in the eye data to determine the non-viewing state; or, determine the non-viewing state according to whether the pupil position is located in the eye image included in the eye data; or, determine the non-viewing state according to the difference between the pupil position in the eye image included in the eye data and a preset standard position; If it is determined that the user is not in a viewing state, the screen position is adjusted to the initial position; the adjustment amount of adjusting the screen position to the initial position is 1 / 10°; After the screen position is adjusted to the initial position, the method further includes: within a preset time interval, no longer collecting the user's eye data.

2. A screen position correction device, applied to a head mounted display device, It is characterized in that include: A collection component, used to collect eye data of the user; A control component, used for determining whether the user is in a non-viewing state according to the eye data; If it is determined that the user is not in a viewing state, the screen position is adjusted to an initial position; after the screen position is adjusted to the initial position, the user's eye data is no longer collected within a preset time interval; wherein the adjustment amount of adjusting the screen position to the initial position is 1 / 10°; Among them, judging whether the user blinks according to the electrical pulses of the periocular muscles included in the eye data to determine the non-viewing state; or, judging whether the pupil position is located in the eye image included in the eye data to determine the non-viewing state; or, judging whether the pupil position in the eye image included in the eye data is compared with a preset standard position to determine the non-viewing state.

3. The picture position correction device according to claim 2, It is characterized in that The acquisition component is an eye movement sensor; the eye movement sensor is arranged at the position where the head-mounted display device contacts the user's eyes.

4. The picture position correction device according to claim 2, It is characterized in that The acquisition component is an infrared camera arranged inside the head-mounted display device.

5. The picture position correction device according to claim 4, It is characterized in that Also includes: At least one infrared lamp is arranged inside the head mounted display device.

6. A head mounted display device, It is characterized in that include: A picture position correction device as claimed in any one of claims 2 to 5.

Images