VR device with eye tracking function and eye tracking method

Abstract

The present invention provides a VR device with an eye tracking function and an eye tracking method. The VR device with an eye tracking function includes: a VR device body, an optical module, a display screen, an infrared fill light and an infrared camera; the infrared fill light is arranged on the VR device body, and is used to emit infrared light to fill the eyes of the user wearing the VR device. The optical module and the display screen are arranged in sequence along the user's line of sight, so that the optical module amplifies the image displayed on the display screen and displays the amplified image to the user. The infrared camera is arranged between the optical module and the display screen, so as to capture the user's eyes through the optical module to obtain an infrared image of the eyes, so as to realize eye tracking of the user based on the infrared image of the eyes and control the VR device based on the result of eye tracking. The infrared camera of the present invention has a small tilt angle, which can simplify the calculation difficulty of eye tracking, improve the calculation accuracy, and amplify the eyes through the optical module to reduce calculation errors.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to a VR device with an eye tracking function and an eye tracking method. Background Art

[0002] The human eye consists of three parts: the sclera, the iris, and the pupil. The iris is located between the sclera and the pupil. Its texture is complex, with numerous features, making it difficult to forge. Iris biometrics technology offers uniqueness, stability, security, non-contact, and high capacity. Therefore, iris recognition is more secure than fingerprint and facial recognition, and is one of the most accurate and forgery-resistant biometric technologies currently available. Eye tracking, the process of identifying where and how someone is looking, has been widely used in fields such as human-computer interaction, virtual reality, assisted driving, human factors analysis, and psychological research.

[0003] However, the various devices of existing eye tracking equipment are independent of each other, and the camera is obviously tilted. The human eye is not facing the camera in the image captured, resulting in a serious tilt in the shooting angle, which is not convenient for eye tracking. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a VR device with eye tracking function and an eye tracking method, which can effectively simplify the calculation difficulty of eye tracking, improve the calculation accuracy, and reduce the advantages of calculation error.

[0005] The specific technical solutions are as follows:

[0006] A VR device with an eye tracking function comprises: a VR device body, an optical module, a display screen, an infrared fill light, and an infrared camera; the infrared fill light is arranged on the VR device body and is used to emit infrared light to fill the eyes of a user wearing the VR device;

[0007] The optical machine module and the display screen are sequentially arranged along the line of sight of the user, so as to amplify the image displayed on the display screen through the optical machine module and display the amplified image to the user;

[0008] The infrared camera is arranged between the optical module and the display screen to capture the user's eyes through the optical module to obtain an infrared image of the eyes, so as to track the user's eye movements based on the infrared image of the eyes and control the VR device based on the results of the eye tracking.

[0009] In a specific embodiment, the optical machine module includes one of a convex lens, a Fresnel lens or a PanCake optical machine module.

[0010] In a specific embodiment, the edge of the display screen forms a plurality of optical paths from the edge of the display screen to the eye through the optical mechanical module; the surface where the optical path from the edge of the display screen to the optical mechanical module lies is an incident surface, and the surface where the optical path from the optical mechanical module to the eye lies is a refractive surface;

[0011] There is a preset angle between the incident surface and the extension surface of the refractive surface, and the infrared camera is arranged between the incident surface and the extension surface of the refractive surface.

[0012] In a specific embodiment, the infrared camera is disposed in a region close to the incident surface.

[0013] In a specific embodiment, the center of the optical engine module, the center of the display screen and the user's eyes are located on the same straight line.

[0014] In a specific embodiment, the infrared camera is located obliquely below the optical machine module.

[0015] In a specific embodiment, the infrared camera is located obliquely above the optical machine module.

[0016] In a specific embodiment, the infrared fill light includes at least one red LED lamp.

[0017] In a specific embodiment, the VR device body includes an eye tracking calculation unit and a main control board. The eye tracking calculation unit is electrically connected to the main control board and is used to determine the eye tracking result based on the eye infrared image and send the eye tracking result to the main control board.

[0018] An eye tracking method is applied to the above-mentioned VR device with eye tracking function, the method comprising:

[0019] Starting the infrared fill light to emit infrared light, and controlling the infrared camera to shoot, so as to obtain the infrared image of the eye;

[0020] performing image processing on the eye infrared image to determine an eye tracking result;

[0021] The VR device is controlled based on the eye tracking result.

[0022] The present invention has at least the following beneficial effects:

[0023] The present invention provides a VR device with an eye tracking function and an eye tracking method. The VR device with an eye tracking function includes: a VR device body, an optical module, a display screen, an infrared fill light and an infrared camera; the infrared fill light is arranged on the VR device body, and is used to send infrared light to fill the eyes of the user wearing the VR device. The optical module and the display screen are arranged in sequence along the user's line of sight, so that the optical module amplifies the image displayed on the display screen and displays the amplified image to the user. The infrared camera is arranged between the optical module and the display screen, so as to capture the user's eyes through the optical module to obtain an infrared image of the eyes, so as to realize eye tracking of the user based on the infrared image of the eyes and control the VR device based on the result of eye tracking. The infrared camera of the present invention has a small tilt angle, which can effectively simplify the calculation difficulty of eye tracking and improve the calculation accuracy. Moreover, by amplifying the eyes through the optical module, the calculation error can be further reduced and the calculation accuracy can be improved.

[0024] Furthermore, the surface where the light path from the edge of the display screen to the optical module is located is the incident surface, and the surface where the light path from the optical module to the eye is located is the refractive surface. By setting the infrared camera between the incident surface and the extension surface of the refractive surface, it is possible to avoid the camera blocking the light path from the edge of the display screen to the optical module, and at the same time, the tilt angle of the infrared camera can be reduced to improve the accuracy of eye tracking.

[0025] Furthermore, the infrared camera is arranged in an area close to the incident surface to better reduce the tilt angle of the infrared camera and thus improve the accuracy of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 A schematic diagram of the optical path of a VR device with eye tracking function provided by the present invention;

[0028] Figure 2 A schematic diagram of a VR device with eye tracking function provided by the present invention;

[0029] Figure 3 A stereoscopic diagram of the optical path of a VR device with eye tracking function provided by the present invention;

[0030] Figure 4 A schematic diagram of the eye tracking method provided by the present invention;

[0031] Figure 5 Schematic diagram of the PanCake optical-mechanical module of the VR device with eye tracking function provided by the present invention.

[0032] Reference numerals:

[0033] 1- VR device body; 2- optical machine module; 3- display screen; 4- infrared fill light; 5- infrared camera;

[0034] 6-Incident surface; 7-Refractive surface. DETAILED DESCRIPTION

[0035] Hereinafter, various embodiments of the present invention will be described more fully. The present invention can have various embodiments, and modifications and variations can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present invention to the specific embodiments disclosed herein, but rather that the present invention should be construed to encompass all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the present invention.

[0036] Hereinafter, the terms "include" or "may include" used in various embodiments of the present invention indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present invention, the terms "include", "have" and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be understood as excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0037] In various embodiments of the present invention, the expression "or" or "at least one of A or / and B" includes any or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0038] The expressions (such as "first", "second", etc.) used in the various embodiments of the present invention may modify the various constituent elements in the various embodiments, but may not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used to distinguish one element from other elements. For example, a first user device and a second user device indicate different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of the present invention, a first element may be referred to as a second element, and similarly, a second element may also be referred to as a first element.

[0039] It should be noted that, in the present invention, unless otherwise expressly specified or defined, terms such as "mounted," "connected," and "fixed" should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in the present invention based on specific circumstances.

[0040] In the present invention, those skilled in the art need to understand that the terms indicating orientation or positional relationships herein are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0041] The terms used in various embodiments of the present invention are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments of the present invention. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise limited, all terms used here (including technical terms and scientific terms) have the same meaning as those of ordinary skill in the art generally understood by the various embodiments of the present invention. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having idealized meaning or too formal meaning, unless clearly defined in various embodiments of the present invention.

[0042] Example 1

[0043] like Figure 1-3 As shown, the present invention provides a VR device with an eye tracking function, comprising: a VR device body 1, an optical module 2, a display screen 3, an infrared fill light 4 and an infrared camera 5; the infrared fill light 4 is arranged on the VR device body 1, and is used to send infrared light to fill the eyes of the user wearing the VR device. The optical module 2 and the display screen 3 are arranged in sequence along the user's line of sight, so that the image displayed on the display screen 3 is amplified by the optical module 2 and the amplified image is displayed to the user. The infrared camera 5 is arranged between the optical module 2 and the display screen 3, so as to capture the user's eyes through the optical module 2 to obtain an infrared image of the eyes, so as to realize eye tracking of the user based on the infrared image of the eyes and control the VR device based on the result of eye tracking.

[0044] The present invention places an infrared camera 5 between the optical module 2 and the display screen 3 to capture the user's eyes through the optical module 2 to obtain an infrared image of the eyes. Compared to existing technologies, the infrared camera 5 has a smaller tilt angle, making the captured infrared image easier to track eye movements, simplifying calculations and improving accuracy. Furthermore, the optical module 2 magnifies the eyes, resulting in a clearer infrared image of the eyes, effectively reducing calculation errors and improving accuracy.

[0045] Specifically, the optical module 2, display screen 3, infrared fill light 4, and infrared camera 5 are all installed on the VR device body 1. The optical module 2 and display screen 3 are arranged in sequence along the user's line of sight. Display screen 3 is used to display screen content and emit light. The optical module 2 is used to receive light emitted by display screen 3 and change the light path. The infrared fill light 4 is used to emit infrared light to fill the eyes of the user wearing the VR device.

[0046] like Figure 2 、 3 As shown, the edge of the display screen passes through the optical module 2 to the eye to form a plurality of optical paths; the surface where the optical path from the edge of the display screen to the optical module 2 is located is the incident surface 6, and the surface where the optical path from the optical module 2 to the eye is located is the refractive surface 7;

[0047] There is a preset angle between the incident surface 6 and the extension of the refractive surface 7, and the infrared camera 5 is disposed between the incident surface 6 and the extension of the refractive surface 7. By disposing the infrared camera 5 between the incident surface 6 and the extension of the refractive surface 7, the camera is prevented from blocking the optical path from the edge of the display screen to the optical engine module 2, and the tilt angle of the infrared camera 5 is reduced, thereby improving the accuracy of eye tracking.

[0048] Specifically, if Figure 2 As shown, both the incident surface 6 and the refractive surface 7 are conical columns, and the preset angle between the incident surface 6 and the extension of the refractive surface 7 is an acute angle. There can be one or more infrared cameras 5, and all infrared cameras 5 are arranged between the incident surface 6 and the extension of the refractive surface 7.

[0049] In one embodiment, there is one infrared camera 5 , which is disposed between the incident surface 6 and the extension surface of the refractive surface 7 .

[0050] In other embodiments, there are multiple infrared cameras 5, all of which are disposed between the incident surface 6 and the extension of the refractive surface 7. The multiple infrared cameras 5 capture a large number of eye infrared images, thereby facilitating subsequent eye tracking of the user based on the eye infrared images and improving calculation accuracy.

[0051] like Figure 1-3As shown, the infrared camera 5 is arranged in an area close to the incident surface 6. By arranging the infrared camera 5 in an area close to the incident surface 6, the tilt angle of the infrared camera 5 can be better reduced, thereby improving the accuracy of eye tracking.

[0052] The infrared camera 5 is disposed between the optical module 2 and the display screen 3 , and may be located in an area close to the optical module 2 or in an area close to the display screen 3 .

[0053] In one embodiment, the infrared camera 5 is disposed between the optical-mechanical module 2 and the display screen 3 , and is close to the incident surface 6 and the optical-mechanical module 2 .

[0054] In another embodiment, the infrared camera 5 is disposed between the optical machine module 2 and the display screen 3 , and is close to the incident surface 6 and the display screen 3 .

[0055] like Figure 1-3 As shown, the center of the optical module 2, the center of the display screen 3, and the user's eyes are aligned. Specifically, the line connecting the three centers of the optical module 2, the display screen 3, and the user's eyes is the center line. An infrared camera 5 is positioned around the center line, between the optical module 2 and the display screen 3.

[0056] In one embodiment, the infrared camera 5 is disposed between the optical machine module 2 and the display screen 3 , and the infrared camera 5 is located obliquely below the optical machine module 2 .

[0057] In another embodiment, the infrared camera 5 is disposed between the optical-mechanical module 2 and the display screen 3 , and is located obliquely above the optical-mechanical module 2 .

[0058] In other embodiments, the infrared camera 5 is disposed between the optical-mechanical module 2 and the display screen 3 , and is located obliquely to the side of the optical-mechanical module 2 .

[0059] Preferably, setting the infrared camera 5 diagonally below the optical module 2 enhances the stability of the VR device structure with eye tracking function. At the same time, setting the infrared camera 5 diagonally below the optical module 2 can reduce the gravity pressure exerted by the infrared camera 5 on the user, thereby improving the user's experience.

[0060] like Figure 1-3 As shown, the optical module 2 includes one of a convex lens, a Fresnel lens, or a PanCake optical module. Each of these can alter the optical path of the display screen 3 as it passes through the optical module 2 and reaches the user's eyes, thereby allowing the user to see the image displayed on the magnified display screen 3.

[0061] Preferably, Figure 5As shown, optical module 2 uses a PanCake optical module. The PanCake optical module includes a first lens and a second lens. One side of the first lens is provided with a transflective film, the other side is provided with a wave plate, and the second lens is provided with a polarizing beam splitter film. Light from the display screen passes through the first lens, wave plate, polarizing beam splitter film, and second lens before reaching the human eye. Compared to other optical modules such as convex lenses and Fresnel lenses, the PanCake optical module has the advantages of being smaller, lighter, and more compact.

[0062] Among them, such as Figure 2 、 3 As shown, the infrared fill light 4 includes at least one red LED, which is arranged on the display screen 3 and is used to emit infrared light to fill in the eyes of the user wearing the VR device. The red LED is arranged on the display screen 3 so that the emitted infrared light is directly facing the user's eyes to prevent insufficient fill light from causing unclear infrared images of the eyes.

[0063] The VR device body 1 includes an eye tracking calculation unit and a main control board. The eye tracking calculation unit is electrically connected to the main control board for determining the eye tracking result based on the eye infrared image and sending the eye tracking result to the main control board.

[0064] Specifically, the VR device body 1 also includes an iris recognition unit, which is electrically connected to the main control board and is used to perform iris recognition on the infrared image of the eye and send the recognition result to the main control board.

[0065] Example 2

[0066] like Figure 4 As shown, an eye tracking method is applied to a VR device with an eye tracking function, the method comprising:

[0067] S1: Start the infrared fill light to emit infrared light, and control the infrared camera to shoot to obtain an infrared image of the eye.

[0068] S2: Perform image processing on the eye infrared image to determine the eye tracking result.

[0069] S3: Control the VR device based on the results of eye tracking.

[0070] The eye tracking method of the present invention has the advantages of simplifying calculation difficulty, improving calculation accuracy and reducing calculation errors.

[0071] Specifically, there are one or more infrared fill lights and infrared cameras.

[0072] In one embodiment, there are multiple infrared fill lights and infrared cameras, and the multiple infrared cameras are located at different positions; S1 "starting the infrared fill light to emit infrared light, and controlling the infrared camera to shoot at the same time to obtain infrared images of the eyes" includes: starting multiple infrared fill lights to emit infrared light, and controlling multiple infrared cameras to shoot at the same time to obtain infrared images of the eyes at multiple different angles.

[0073] In another embodiment, there is one infrared camera, and S1 "starting the infrared fill light to emit infrared light, and controlling the infrared camera to shoot to obtain an infrared image of the eye" includes: starting the infrared fill light to emit infrared light, and controlling the infrared camera to shoot to obtain an infrared image of the eye.

[0074] S2 “performing image processing on the infrared image of the eye to determine the result of eye tracking” includes: establishing a coordinate system, obtaining the coordinates of each feature point of the infrared image of the eye, performing image processing based on the coordinates of the feature points to obtain the result of image processing; and determining the result of eye tracking based on the result of image processing.

[0075] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0076] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A VR device with eye tracking function, characterized in that , including: a VR device body, an optical machine module, a display screen, an infrared fill light and an infrared camera; the infrared fill light is set on the VR device body, and is used to send infrared light to fill the eyes of the user wearing the VR device; The optical machine module and the display screen are sequentially arranged along the line of sight of the user, so as to amplify the image displayed on the display screen through the optical machine module and display the amplified image to the user; The infrared camera is arranged between the optical module and the display screen to capture the user's eyes through the optical module to obtain an infrared image of the eyes, so as to track the user's eye movements based on the infrared image of the eyes and control the VR device based on the result of the eye tracking; The optical machine module includes one of a convex lens, a Fresnel lens or a PanCake optical machine module; The edge of the display screen passes through the optical mechanism module to reach the eye, forming a plurality of optical paths; the surface where the optical path from the edge of the display screen to the optical mechanism module lies is the incident surface, and the surface where the optical path from the optical mechanism module to the eye lies is the refractive surface; There is a preset angle between the incident surface and the extension surface of the refractive surface, and the infrared camera is arranged between the incident surface and the extension surface of the refractive surface.

2. The VR device with eye tracking function according to claim 1, characterized in that: The infrared camera is arranged in a region close to the incident surface.

3. The VR device with eye tracking function according to claim 1, characterized in that , the center of the optical module, the center of the display screen and the user's eyes are located on the same straight line.

4. The VR device with eye tracking function according to claim 1, characterized in that , the infrared camera is located obliquely below the optical module.

5. The VR device with eye tracking function according to claim 1, characterized in that , the infrared camera is located obliquely above the optical module.

6. The VR device with eye tracking function according to claim 1, characterized in that , the infrared fill light includes at least one red LED light.

7. The VR device with eye tracking function according to claim 1, characterized in that The VR device body includes an eye tracking calculation unit and a main control board. The eye tracking calculation unit is electrically connected to the main control board and is used to determine the eye tracking result based on the eye infrared image and send the eye tracking result to the main control board.

8. An eye tracking method, applied to the VR device with eye tracking function according to any one of claims 1 to 7, characterized in that: The method includes: Starting the infrared fill light to emit infrared light, and controlling the infrared camera to shoot, so as to obtain the infrared image of the eye; performing image processing on the eye infrared image to determine an eye tracking result; The VR device is controlled based on the eye tracking result.

Images