Eye tracking apparatus, eye tracking method and head-mounted display device

Abstract

An eye tracking apparatus, an eye tracking method and a head-mounted display device. The eye tracking apparatus comprises: a frame (100); an optical display module (200, 300), which is arranged on the frame (100); a light source (500), which is arranged on the frame (100), wherein light of the light source (500) can be reflected by eyeballs (400); a reflective layer (600), which is arranged on the optical display module (200, 300) and is used for reflecting the light emitted by the light source (500) and reflected by the eyeballs (400); a light sensor (700), which is arranged on the frame (100) and is used for receiving the light reflected by the reflective layer (600) and / or the light of the light source (500) reflected by the eyeballs (400) so as to obtain light sensing data, and transmitting the light sensing data to a processor; and the processor, which is arranged on the frame (100), receives the light sensing data from the light sensor (700), obtains light intensity data on the basis of the light sensing data, and determines the rotational angle of the eyeballs (400) on the basis of the light intensity data and a preset mapping relationship. The eye tracking apparatus can overcome the problem of the eye relief in an eye tracking apparatus being overly small, such that the position of the eyeballs (400) can be accurately tracked, and the field of view (FOV) can be expanded.

Description

Eye tracking device, eye tracking method and head-mounted display device

[0001] Related applications

[0002] This invention claims priority of the Chinese invention patent with patent application number 202410370544.0, application date 2024-03-28, and invention name “Eye tracking device, eye tracking method and head-mounted display device”. Technical Field

[0003] The present invention relates to the field of display technology, and in particular to an eye tracking device, an eye tracking method and a head-mounted display device. Background Art

[0004] Currently, augmented reality (AR) and / or virtual reality (VR) systems can generate three-dimensional (3D) immersive environments. Users can interact with 3D virtual immersive environments through various electronic devices to experience them. For example, MR glasses or other near-eye display devices (including VR glasses, AR glasses, XR glasses, etc.) are collectively referred to as head-mounted displays (HMDs). However, due to optical design reasons, the eye relief distance is getting smaller and smaller. However, the reduced eye relief distance will greatly reduce the field of view (FOV) of the eye tracking system sensor, which in turn affects the eye tracking accuracy and tracking speed. In addition, existing eye tracking devices have high power consumption. Summary of the Invention

[0005] The present invention aims to at least partially address one of the technical problems in the related art. To this end, one object of the present invention is to provide an eye tracking device that can effectively and accurately track the eyeball, expand the field of view, and reduce power consumption under the condition of a small pupil distance.

[0006] In one aspect, the present invention provides an eye tracking device. According to an embodiment of the present invention, the eye tracking device includes: a frame; an optical display module, the optical display module being disposed on the frame; at least one light source, the light source being disposed on the frame, and the light emitted by the light source being reflective of the eyeball; at least one reflective layer, the reflective layer being disposed on the optical display module and being used to reflect the light emitted by the light source reflected by the eyeball; at least one light sensor, the light sensor being disposed on the frame and being used to receive the light reflected by the reflective layer and / or the light from the light source reflected by the eyeball, to obtain light sensing data, and to transmit the light sensing data to a processor; a processor, the processor being disposed on the frame and being used to receive the light sensing data from the at least one light sensor, obtain light intensity data based on the light sensing data, and determine an eye rotation angle based on the light intensity data and a preset mapping relationship, wherein the preset mapping relationship represents the relationship between light intensity and eye rotation angle.

[0007] Therefore, in the eye tracking device of the above structure, the light emitted by the light source is incident on the eyeball, and the eyeball reflects at least part of the light to the reflective layer. The reflective layer reflects the light reflected by the eyeball again and is received by the light sensor. At the same time, the light sensor can further receive the light emitted by the light source that is directly reflected by the eyeball. The light sensor obtains light sensing data based on the received reflected light, and the processor obtains light intensity data based on the light sensing data. Based on the light intensity data and a preset mapping relationship, the rotation angle of the eyeball can be determined, thereby realizing eye tracking. At the same time, due to the setting of the reflective layer, the eye tracking device of the above structure can also overcome the problem of too small exit pupil distance in the eye tracking device, and thus can accurately track the eyeball position, expand the field of view angle and reduce power consumption.

[0008] According to an embodiment of the present invention, when the at least one light sensor includes multiple light sensors, the preset mapping relationship includes multiple relationship curves, and the multiple relationship curves correspond one-to-one to the multiple light sensors; the changing trends of the light intensity data in each relationship curve with the eye rotation angle are not completely consistent.

[0009] According to an embodiment of the present invention, the optical display module includes: a display screen; and a folded light path lens located on a side of the display screen close to the eyeball.

[0010] According to an embodiment of the present invention, the setting position of the reflective layer satisfies at least one of the following conditions: the reflective layer is set on the surface of the display screen close to the folding light path lens; the reflective layer is set between the display screen and the folding light path lens; the reflective layer is set on the surface of the folding light path lens close to the display screen; the reflective layer is set on the surface of the folding light path lens away from the display screen.

[0011] According to an embodiment of the present invention, the light sensor includes at least one of the following: a shooting camera, wherein the light sensing data includes image data, and the processor is used to sum multiple pixel values ​​in the image data to obtain a pixel sum value, and use the pixel sum value as the light intensity data; a photodetector, wherein the light sensing data includes a current signal, and the processor is used to obtain the light intensity data based on the current signal.

[0012] According to an embodiment of the present invention, the preset mapping relationship includes a mapping function, which takes the eye rotation angle as the independent variable and the light intensity as the dependent variable; determining the eye rotation angle based on the light intensity data and the preset mapping relationship includes: inputting the light intensity data as the dependent variable into the mapping function, and outputting the eye rotation angle.

[0013] According to an embodiment of the present invention, the preset mapping relationship is generated in the following manner: the processor generates indication information and displays it on the display screen, wherein the display position of the indication information can represent the eye movement angle of the user when looking at the indication information; the user's eyeballs move while looking at the indication information, and light sensing data samples are obtained through the at least one light sensor; the processor determines the eye movement angle samples based on the display position of the indication information, obtains the light intensity data samples based on the light sensing data samples, and fits the preset mapping relationship based on the eye movement angle samples and the light intensity data samples.

[0014] According to an embodiment of the present invention, the processor is also used to: determine whether a preset mapping relationship is stored for the current user; if so, retrieve the stored preset mapping relationship to determine the eye rotation angle based on the retrieved preset mapping relationship; if not, generate the preset mapping relationship for the current user.

[0015] According to an embodiment of the present invention, the light sensor is arranged facing the reflective layer and / or the eyeball, so as to receive light from the reflective layer and / or the eyeball to obtain the light sensing data.

[0016] According to an embodiment of the present invention, the frame includes a connected frame body and a supporting component, the light source and the light sensor are arranged on the supporting component, and / or the light source and the light sensor are arranged on the frame and close to the edge of the folded light path lens.

[0017] According to an embodiment of the present invention, the orthographic projections of the plurality of light sensors on the human face do not overlap.

[0018] According to an embodiment of the present invention, when the at least one light source includes a plurality of the light sources, the plurality of light sources are arranged in multiple rows and columns, and / or the plurality of light sources are dispersed around the edge of the folded light path lens.

[0019] According to an embodiment of the present invention, the light sensor is arranged between the reflective layer and the eyeball and distributed on the edge of the folded light path lens, and / or the light sensor is arranged in the gap between the light sources.

[0020] According to an embodiment of the present invention, the thickness of one layer of the reflective layer is 50-500 nm, and the visible light transmittance of one layer of the reflective layer is greater than or equal to 90%.

[0021] According to an embodiment of the present invention, one layer of the reflective layer includes a plurality of stacked sub-reflective layers.

[0022] According to an embodiment of the present invention, the light source includes an infrared light source.

[0023] In another aspect, the present invention provides an eye tracking method implemented using the aforementioned eye tracking device. According to an embodiment of the present invention, the eye tracking method includes: obtaining light intensity data based on light sensing data, wherein the light sensing data is obtained based on light reflected by the eyeball and / or based on light reflected by a reflective layer; and determining an eye rotation angle based on the light intensity data and a preset mapping relationship, wherein the preset mapping relationship represents the relationship between light intensity and eye rotation angle.

[0024] Thus, light emitted by the light source is incident on the eyeball, which then reflects at least a portion of the light onto the reflective layer. The reflective layer then reflects the light reflected by the eyeball again, which is then received by the light sensor. Simultaneously, the light sensor can also receive light emitted by the light source that is directly reflected by the eyeball. The light sensor obtains light sensing data based on the received reflected light. The processor obtains light intensity data based on the light sensing data and determines the eyeball's rotation angle based on the light intensity data and a preset mapping relationship, thereby achieving eye tracking. Furthermore, due to the provision of the reflective layer, the eye tracking device with the above structure can overcome the problem of a small exit pupil distance in eye tracking devices, thereby accurately tracking the eyeball's position and expanding the field of view (FOV).

[0025] In another aspect, the present invention provides a head-mounted display device. According to an embodiment of the present invention, the head-mounted display device includes the eye tracking device described above or is used to implement the eye tracking method described above. As a result, the head-mounted display device has an appropriate exit pupil distance, accurately tracks eye position, and has a large field of view, thereby effectively improving the interactive experience of the head-mounted display device. Those skilled in the art will understand that the head-mounted display device has all the features and advantages of the eye tracking device described above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0027] FIG1 is a schematic diagram of the structure of an eye tracking device according to an embodiment of the present invention;

[0028] FIG2 is a schematic structural diagram of an eye tracking device according to another embodiment of the present invention;

[0029] FIG3 is a schematic diagram of the structure of an eye tracking device according to another embodiment of the present invention;

[0030] FIG4 is a schematic structural diagram of an eye tracking device according to another embodiment of the present invention;

[0031] FIG5 is a schematic structural diagram of an eye tracking device according to another embodiment of the present invention;

[0032] FIG6 is a schematic structural diagram of an eye tracking device according to another embodiment of the present invention;

[0033] FIG7 is a schematic structural diagram of an eye tracking device according to another embodiment of the present invention;

[0034] FIG8 is a schematic diagram of the positional relationship between a light source and a light sensor in one embodiment of the present invention;

[0035] FIG9 is a schematic diagram of the positional relationship between a light source and a light sensor in another embodiment of the present invention;

[0036] FIG10 is a schematic diagram of a light sensor photographing according to another embodiment of the present invention;

[0037] FIG11A is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0038] FIG11B is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0039] FIG12 is a schematic diagram of a light sensor photographing according to another embodiment of the present invention;

[0040] FIG13A is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0041] FIG13B is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0042] FIG14A is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0043] FIG14B is a graph showing a light sensor-eye rotation angle curve according to another embodiment of the present invention;

[0044] FIG15 is a simulation process for obtaining a light sensor-eye rotation angle curve diagram in another embodiment of the present invention. DETAILED DESCRIPTION

[0045] The scheme of the present invention will be explained below with reference to the examples. Those skilled in the art will understand that the following examples are only used to illustrate the present invention and should not be construed as limiting the scope of the present invention. Where specific techniques or conditions are not specified in the examples, the techniques or conditions described in the literature in this area or the product specifications shall be followed.

[0046] The present invention is described below with reference to specific examples. It should be noted that these examples are merely illustrative and do not limit the present invention in any way.

[0047] In one aspect, the present invention provides an eye-tracking device, which includes wearable devices such as glasses and helmets. For ease of understanding, Figure 1 uses glasses as an example for description. According to an embodiment of the present invention, referring to Figure 1, the eye-tracking device includes: a frame 100, an optical display module (as shown in Figures 200 and 300), at least one light source 500, at least one reflective layer 600, at least one light sensor 700, and a processor. In the case where the eye-tracking device is glasses, the frame 100 includes a glasses frame.

[0048] The optical display module is arranged on the frame 100; the light source 500 is arranged on the frame 100, and the light emitted by the light source 500 can enter the eyeball 400 and be reflected by the eyeball 400; at least one reflective layer 600 (one reflective layer is taken as an example in Figure 1) is arranged on the optical display module, which is used to reflect the light reflected by the eyeball 400; at least one light sensor (sensor) 700 is arranged on the frame 100, which is used to receive the light reflected by the reflective layer 600 and / or the light emitted by the light source reflected by the eyeball 400, obtain light sensing data, and transmit the light sensing data to the processor.

[0049] The processor is set on the frame, and the processor may include a chip for receiving light sensing data from at least one light sensor 700, obtaining light intensity data based on the light sensing data, and determining the eye rotation angle based on the light intensity data and a preset mapping relationship. The preset mapping relationship represents the relationship between light intensity and eye rotation angle. Therefore, in the eye tracking device of the above structure, the light emitted by the light source 500 is incident on the eyeball 400, and the eyeball 400 reflects at least part of the light to the reflective layer 600. The reflective layer 600 reflects the light reflected by the eyeball 400 again and is received by the light sensor 700. At the same time, the light sensor can further receive the light emitted by the light source that is directly reflected by the eyeball. The light sensor 700 obtains light sensing data based on the received reflected light. The processor obtains light intensity data based on the light sensing data, and can determine the rotation angle of the eyeball 400 based on the light intensity data and a preset mapping relationship, thereby realizing eye tracking. At the same time, due to the setting of the reflective layer, the eye tracking device of the above structure can also overcome the problem of too small exit pupil distance in the eye tracking device, and can thus accurately track the eye position and expand the field of view (FOV). In some embodiments, a field of view angle range of 20° can be achieved.

[0050] In another example, the optical display module includes a display screen 200 and a pancake lens 300 , and the display screen 200 and the pancake lens 300 are both arranged on the frame 100 . Furthermore, the pancake lens 300 is located on the side of the display screen 200 close to the eyeball 400 .

[0051] Among them, those skilled in the art can understand that there is no special requirement for the specific type of display screen, and those skilled in the art can flexibly choose according to actual needs, such as LCD display screen, OLED display screen, Micro LED display screen, ULED display screen, etc.

[0052] According to some embodiments of the present invention, there are no special requirements for the specific structure of the folded light path lens (pancakelens) 300, and those skilled in the art can flexibly select it based on actual needs and existing technical means. For example, in some embodiments, the folded light path lens includes a lens, an optical functional layer, and a semi-transparent and semi-reflective film (BS), wherein the optical functional layer is attached to the surface of the lens, and the semi-transparent and semi-reflective film is arranged on the side of the optical functional layer away from the lens, wherein the optical film includes at least one of a wavelength phase delay film (QWP, such as a 1 / 4 wavelength phase delay film, a 1 / 2 wavelength phase delay film, etc.), a reflective polarizing film (PR), and a linear polarizing film (LR). Of course, the above is only one specific structure of the folded light path lens in the present invention, and is not a limitation on the specific structure of the folded light path lens of the present invention.

[0053] According to some embodiments of the present invention, the light source includes an infrared light source. Thus, infrared light can effectively assist the eye tracking device of the present invention in tracking the position of the eyeball. Furthermore, the infrared light source can be an infrared LED (IR LED). Accordingly, the reflective layer is an infrared light reflective layer for reflecting infrared light, and the light sensor is an infrared sensor (IR sensor) for receiving infrared light. The specific type of infrared sensor has specific requirements, as long as it can achieve the above-mentioned technical effects of the present invention.

[0054] In the embodiments of the present invention, the reflective layer needs to be provided on the side of the display screen 200 close to the eyeball (i.e., the user side), including the reflective layer provided on the side of the display screen 200 close to the eyeball, the reflective layer provided on the folded light path lens 300, and the reflective layer provided between the display screen 200 and the folded light path lens 300. Regardless of how the reflective layer is designed, it is necessary to prevent infrared light from entering the display screen. The following describes in detail the location of the reflective layer 600 according to some embodiments of the present invention:

[0055] In some embodiments of the present invention, referring to FIG. 2 , a reflective layer 600 is disposed on the surface of the display screen 200 near the folded light path lens 300. Thus, light emitted by the light source 500 is reflected by the eyeball 400, passes through the folded light path lens 300, and strikes the reflective layer 600. The reflective layer 600 then reflects the light, passing through the folded light path lens 300 and being received by the light sensor 700. In this manner, the reflective layer 600 and the display screen 200 can be mounted on the frame as a single unit, improving the stability of the reflective layer.

[0056] In some embodiments of the present invention, referring to FIG1 and FIG3 , the reflective layer 600 is disposed between the display screen 200 and the folded light path lens 300. This means that the reflective layer 600 need not be disposed on the surface of the display screen 200 near the folded light path lens 300. In other words, the reflective layer 600 need not be disposed in contact with the display screen 200 or the folded light path lens 300, but only needs to be located between the display screen 200 and the folded light path lens 300. In this way, light emitted by the light source 500 is reflected by the eyeball 400, passes through the folded light path lens 300, and illuminates the reflective layer 600. The reflective layer 600 then reflects the light, passes through the folded light path lens 300, and is received by the light sensor 700. This structural arrangement can secure the reflective layer to the frame.

[0057] In other embodiments of the present invention, referring to FIG. 4 , a reflective layer 600 is disposed on the surface of the folded optical path lens 300 near the display screen 200. Thus, light emitted by the light source 500 is reflected by the eyeball 400, passes through the folded optical path lens 300, and strikes the reflective layer 600. The reflective layer 600 then reflects the light, passing through the folded optical path lens 300 and being received by the light sensor 700. In this case, the reflective layer 600 and the folded optical path lens 300 can be mounted as a whole on the frame to improve the stability of the reflective layer.

[0058] In other embodiments of the present invention, referring to Figures 5 and 6 , a reflective layer 600 is disposed on the surface of the folded light path lens 300 away from the display screen 200. In this manner, light emitted by the light source 500 is reflected by the eyeball 400 and directly illuminates the reflective layer 600. It is then reflected by the reflective layer 600 to the light sensor 700. During this light propagation path, the light emitted by the light source does not pass through the folded light path lens 300. In this manner, the reflective layer 600 and the folded light path lens 300 can be mounted on the frame as a single unit, improving the stability of the reflective layer.

[0059] In yet other embodiments of the present invention, the eye tracking device includes multiple reflective layers 600 disposed at different locations. The different reflective layers can be disposed at the locations described in the four aforementioned situations. For example, referring to FIG7 , the eye tracking device includes two reflective layers 600: one reflective layer 600 is disposed on the surface of the display screen near the folded optical path lens 300, and the other reflective layer 600 is disposed on the surface of the folded optical path lens 300 near the eyeball 400 (or, in other words, away from the display screen 200). This can improve the reflectivity of light emitted by the light source on the reflective layer surfaces. Specifically, if some light from the light source passes through the reflective layer disposed on the surface of the folded optical path lens, this light can be reflected by the reflective layer disposed on the surface of the display screen, thereby increasing the emissivity of the light emitted by the light source, i.e., increasing the reception rate of the light by the light sensor. This also increases the reflectivity of the light emitted by the reflective layer to the light emitted by the light source. This can further improve the accuracy and efficiency of eye tracking by the eye tracking device.

[0060] According to the embodiments of the present invention, the various reflective layer configurations described above can all effectively receive infrared light reflected by the eyeball and reflect the infrared light to the light sensor.

[0061] According to an embodiment of the present invention, a reflective layer 600 includes multiple layers of stacked sub-reflective layers. Furthermore, in some embodiments, the reflective layer disposed at the same location may be a single-layer structure or a structure of multiple sub-reflective layers. For example, the reflective layer 600 disposed on the surface of the display screen 200 may be a single-layer or multi-layer structure, and the reflective layer 600 disposed on the surface of the folded light path lens (pancake lens) 300 may be a single-layer or multi-layer structure. Furthermore, if a reflective layer is disposed on both the surface of the display screen and the surface of the folded light path lens, the structures of the reflective layers at the two locations may be the same or different. Those skilled in the art may flexibly select the structure based on actual needs, and this is not a limitation herein.

[0062] According to some embodiments of the present invention, the thickness of a reflective layer is 50 to 500 nm. It should be noted that the thickness of the reflective layer refers to the overall thickness of the reflective layer provided at the same position. For example, if the reflective layer is provided on the surface of a display screen, the thickness of the reflective layer refers to the overall thickness of the reflective layer provided on the surface of the display screen; if the reflective layer is provided on the surface of a folded optical path lens, the thickness of the reflective layer refers to the overall thickness of the reflective layer provided on the surface of the folded optical path lens; if reflective layers are provided on both the surface of the display screen and the surface of the folded optical path lens, the thicknesses of the two reflective layers are independently 50 to 500 nm, and the thicknesses of the two reflective layers can be the same or different. Those skilled in the art can flexibly select according to actual needs, and no limitation is imposed herein.

[0063] According to some embodiments of the present invention, the reflective layer has a visible light transmittance of greater than or equal to 90%. The reflective layer 600 has a high visible light transmittance, which does not affect the viewer's viewing quality and experience, nor does it affect the display quality of the display screen. If the reflective layer has a low visible light transmittance, light emitted by the display screen will also be reflected by the reflective layer and will not enter the viewer's eyes, thus affecting the viewer's viewing experience.

[0064] According to some embodiments of the present invention, the light sensor is a camera (such as an infrared camera) or a photodetector. Using an infrared camera can effectively receive light while using a lower pixel count, while using a photodetector can reduce costs, power consumption, and data transmission bandwidth.

[0065] In the case where the light sensor is a camera, the light sensing data includes image data, and the processor is configured to sum multiple pixel values ​​in the image data to obtain a pixel sum value, and use the pixel sum value as the light intensity data. For example, the camera is directed to the eyeball or the reflective layer to obtain image data, and the image data is sent to the processor. The processor processes the image data, for example, by summing multiple pixel values ​​in the image data to obtain a pixel sum value, and using the pixel sum value as the light intensity data. Each shooting moment corresponds to an eyeball rotation angle. When there are multiple cameras, at each shooting moment, each camera shoots an image. In one example, all pixel values ​​in the image can be summed, or multiple pixel values ​​in the local area of ​​the image where the eyeball is located can be summed to obtain a pixel sum value, and the pixel sum value can be used as the light intensity data for the camera.

[0066] When the light sensor is a photodetector, the light sensing data includes a current signal, and the processor is configured to obtain light intensity data based on the current signal. For example, the photodetector detects light from the eyeball or reflective layer to obtain a current signal that can represent light intensity. The current signal is then transmitted to the processor, which is configured to obtain light intensity data based on the current signal. For example, the processor converts the received current signal into a pixel value and uses the pixel value as the light intensity data. When multiple photodetectors are provided, each photodetector performs detection to obtain a current signal specific to that photodetector.

[0067] According to some embodiments of the present invention, the preset mapping relationship includes a mapping function, wherein the mapping function uses eye rotation angle as an independent variable and light intensity as a dependent variable. After obtaining the current light intensity data, the processor may determine the eye rotation angle based on the light intensity data and the preset mapping relationship, for example, by inputting the current light intensity data as the dependent variable into the mapping function and outputting the eye rotation angle.

[0068] According to some embodiments of the present invention, a preset mapping relationship can be generated in advance. For example, the processor generates indication information and displays it on the display screen to instruct the user to look at the indication information. The indication information can be a pattern. The display position of the indication information changes at any time according to the preset display line. When the user's eyes track the indication information, the eyes will move along with the indication information. The display position of the indication information can represent the eye movement angle of the user when looking at the indication information.

[0069] When the user gazes at the indication information, their eyeballs rotate, controlling the light source to emit light so that the light enters the eyeball. The eyeball reflects at least a portion of the light onto the reflective layer, and light sensing data samples are obtained via at least one light sensor. For example, the light sensor is controlled to receive light reflected by the eyeball and / or light reflected by the reflective layer to obtain light sensing data samples, which are then transmitted to the processor. The processor determines eye rotation angle samples based on the display position of the indication information, obtains light intensity data samples based on the light sensing data samples, and generates a preset mapping relationship based on the eye rotation angle samples and the light intensity data samples.

[0070] According to some embodiments of the present invention, the light sensor is positioned facing the reflective layer and / or the eyeball, so as to receive light from the reflective layer and / or the eyeball to obtain the light sensing data. For example, when the light sensor is a camera, the camera's shooting direction is positioned facing the reflective layer and / or the eyeball. This allows for better light reception by the light sensor.

[0071] According to an embodiment of the present invention, the light source and the light sensor may be disposed on a frame, that is, the light source and the light sensor are disposed on the frame and located at the edge of the folded light path lens, and do not block the folded light path lens. The following describes in detail the location of the light source and the light sensor:

[0072] According to some embodiments of the present invention, referring to Figures 1 and 6, a frame 100 includes a connected frame body 110 and a support member 120, with a light source 500 and a light sensor 700 disposed on the support member 120. Taking glasses as an example, the frame body 110 includes the frame body, and the support member 120 includes the temples. In some specific embodiments, the light source 500 and the light sensor 700 can be disposed on a single support member 120 on the same side; in other embodiments, the light source 500 and the light sensor 700 are disposed on two support members on either side, respectively. In other words, at least one light source 500 and at least one light sensor 700 are independently disposed on each of the two support members on either side. Both of these configurations can better enable the light sensor 700 to receive emitted light.

[0073] According to other embodiments of the present invention, referring to FIG8 , the light source 500 and the light sensor 700 are arranged on the frame and close to the edge of the folded light path lens 300. It should be noted that the light source 500 and the light sensor 700 can be arranged at the edge within the folded light path lens area, as shown in (a) in FIG8 ; or the light source 500 and the light sensor 700 can be arranged at the edge outside the folded light path lens 300 area, as shown in (b) in FIG8 ; or part of the light source 500 and / or part of the light sensor 700 can be arranged at the edge within the folded light path lens 300 area, and another part of the light source 500 and / or part of the light sensor 700 can be arranged at the edge outside the folded light path lens 300 area. Those skilled in the art can flexibly choose according to actual conditions. In addition, the number of light sources and light sensors in FIG8 is only for illustration and does not limit the number of light sources and light sensors.

[0074] According to some embodiments of the present invention, as shown in FIG9 , the eye tracking device includes a plurality of light sources 500, which are arranged in multiple rows and columns. FIG9 takes the light sources as an example, which are arranged in 4 rows and 8 columns, and 4 light sensors are arranged in the gaps between the light sources 500. According to other embodiments of the present invention, as shown in FIG8 , the plurality of light sources 500 are dispersed around the edge of the folded optical path lens. The above-mentioned arrangement of light sources and light sensors can effectively improve the determination of the rotation angle of the eye 400, thereby achieving eye tracking, effectively overcoming the problem of too small exit pupil distance in the eye tracking device, and thus can accurately track the eye position and expand the field of view (FOV).

[0075] According to some embodiments of the present invention, referring to Figures 8 and 9 , light sensors are disposed between the reflective layer and the eyeball and distributed at the edge of the folded light path lens, and / or light sensors 700 are disposed in the gaps between the light sources 500. In this way, the light sensors can better capture the light reflected by the reflective layer and the light emitted by the light source directly reflected by the eyeball.

[0076] According to an embodiment of the present invention, when the at least one light sensor includes multiple light sensors, the preset mapping relationship includes multiple relationship curves, each corresponding to the multiple light sensors. The light intensity data in each relationship curve varies in a non-uniform manner with eye rotation angle. Consequently, the correlation between the multiple relationship curves is poor. At the same moment, each light sensor corresponds to the same eye angle. The more light sensors there are, the greater the number of eye angles corresponding to the same position at the same moment, and the more reliable the evidence that the eye was positioned at that angle at that moment.

[0077] According to the embodiments of the present invention, the specific arrangement of the light source and the light source sensor is described in detail above. However, the specific arrangement position and arrangement of the light source and the light sensor can be selected by testing the light intensity data (including normalized light intensity data) and the eye rotation angle of each light sensor to obtain a light intensity data-eye rotation angle curve. By comparing the correlation between the light intensity data-eye rotation angle curves of multiple light sensors, if the correlation between the light intensity data-eye rotation angle curves of different light sensors is poor, it indicates that multiple light sensors in different positions can more comprehensively capture the reflected light, thereby better achieving eye tracking. The following is an explanation with some specific embodiments:

[0078] In some embodiments, the light source and light sensor are arranged as shown in FIG9 , and are both mounted on the support member 120 (the temple of the glasses). A single modular module is installed (located between the eye and the ear). The module includes 8×4 infrared light sources arranged horizontally and vertically, facing the infrared reflective layer; four infrared light sensors are placed between the infrared light source arrays, facing the reflective mirror surface. Taking a camera as an example, the camera captures images of the eyeball at different rotation directions (as shown in FIG10 , where the bright spot in the eyeball in FIG10 is the captured light, and the eyeball rotates from left to right in the image). In the simulation, taking four cameras as an example, each camera captures an image at a certain moment, and all pixel values ​​in the image are added together to obtain a pixel sum value. For a single camera, multiple shots at multiple moments result in multiple pixel sum values, and each of the multiple pixel sum values ​​corresponds to an eyeball rotation angle, thereby forming a light intensity data (pixel sum value)-eyeball rotation angle curve for that camera. Therefore, four signal curves corresponding to the four cameras are generated, such as the four light intensity data (pixel sum value)-eye rotation angle curves shown in Figures 11A and 11B, where the contents of Figures 11A and 11B are the same. Figure 11A is a grayscale solid line curve. In order to more clearly distinguish the solid lines of different grayscales in Figure 11A, Figure 11B uses solid lines, dashed lines and dotted lines of different grayscales to represent the solid lines of different grayscales in Figure 11A.

[0079] Taking four photodetectors as an example, each photodetector collects light at different eye rotation angles to obtain a current signal. The processor converts the current signal into a pixel value, and uses the pixel value as the light intensity data, thereby obtaining four signal curves corresponding to the four photodetectors, such as the light intensity data (pixel value)-eye rotation angle curves shown in Figures 11A and 11B.

[0080] As shown in Figures 11A and 11B, the light intensity data-eye angle curves formed by the four light sensors are relatively consistent. In other words, the correlation between the four light sensors in this case is relatively large. The reason is that the four light sensors are in a linear arrangement and the eye information collected is roughly the same. As a result, the eye information obtained by each light sensor at the same time is relatively consistent, so the curve trends of the four light sensors are roughly the same.

[0081] In other embodiments, the light source and light sensor are arranged as shown in FIG8(a). Specifically, the light source and light sensor are configured as follows: on the nose side: three light sensors are placed (i.e., the position of the light sensor at the folded optical path lens corresponds to the nose side of the face), of which one (numbered 8) faces the reflective layer for information collection, and the collected information includes an image or current signal; on the front upper eyelid: one light sensor (numbered 7, i.e., the position of the light sensor at the folded optical path lens corresponds to the front upper upper eyelid of the face), and the information collection direction of this light sensor faces the reflective layer; on the cheek side: four light sensors are placed (i.e., the position of the four light sensors at the folded optical path lens corresponds to the cheek side of the face), of which one (numbered 4) faces the reflective layer for information collection; and other light sensors: light sensors 1, 2, 3, 5, and 6 face the eyeball directly for information collection. A schematic diagram of information collection can be seen in FIG12. Among them, light sensors in poor information collection positions rely on the effect of the reflective film to obtain eyeball information, while light sensors in better positions directly collect eyeball information. Each light sensor is independent of each other. At a certain moment, the light intensity corresponding to the pixel sum value obtained by the shooting camera or the current detected by the photodetector at the current moment can be easily obtained through the characteristics of the light sensor itself (the light intensity is proportional to the pixel value). Through the curve graph (shown in Figures 13A and 13B), the eye rotation angle corresponding to the current light intensity data (the pixel sum value corresponding to the shooting camera or the pixel value corresponding to the photodetector) can be found. That is, by combining the actual value with the theoretical curve graph, the corresponding eye rotation angle can be found, thereby realizing eye tracking. Among them, the contents of Figures 13A and 13B are the same. Figure 13A is a grayscale solid line curve graph. In order to more clearly distinguish the solid lines of different grayscales in Figure 13A, Figure 13B uses solid lines, dashed lines and dotted lines of different grayscales to represent the solid lines of different grayscales in Figure 13A.

[0082] In some other embodiments, the setting requirements of the light source and light sensor are consistent with those of Figure 12 above, with the difference that all the light sensors are facing the eyeball to collect information. The curve graph of the light intensity data of the light sensor and the eyeball rotation angle obtained can be referred to Figures 14A and 14B, wherein the contents of Figures 14A and 14B are the same, Figure 14A is a grayscale solid line curve graph, and in order to more clearly distinguish the solid lines of different grayscales in Figure 14A, Figure 14B uses solid lines, dashed lines and dotted lines of different grayscales to represent the solid lines of different grayscales in Figure 14A. Among them, light sensor No. 4 directly faces the eyeball to collect information, resulting in strong duplication and correlation with the signals of other light sensors. By comparing with Figures 13A and 13B, it can be seen that light sensor No. 4 faces the reflective layer to collect information, thereby capturing the peak signal when the eyeball rotates to the edge angle. In addition, in Figures 14A and 14B, when light sensors No. 7 and No. 8 face the eyeball to collect information, due to the remote viewing angle, the eye information collection area and reflected light signal are relatively small. In Figures 13A and 13B, light sensors No. 7 and No. 8 face the reflective layer to collect information, thereby obtaining more effective signals. It can be seen that the setting position of the light sensors in Figures 13A and 13B can better achieve eye tracking and improve the accuracy of eye tracking, thereby better overcoming the problem of too small exit pupil distance in the eye tracking device, thereby accurately tracking the eyeball position and expanding the field of view (FOV).

[0083] As can be seen from the above, when the orthographic projections of multiple light sensors on the face do not overlap, a curve graph with poor correlation can be better obtained, which is more conducive to obtaining a closer eye rotation angle and more accurate eye tracking.

[0084] From the above explanation, it can be seen that at the same moment, each light sensor collects information and obtains its own light intensity data. The curve of the light intensity data obtained by light sensor 700 and the rotation angle of eyeball 400 can be used to find the eye angle corresponding to each light intensity data. At the same moment, the eye angle corresponding to each light sensor is the same. The more light sensors there are, the more eye angles corresponding to the same position at the same moment, and the more they can be used to prove that the eye is at that angle at that moment. It can also be seen that the eye tracking device of the present invention, combined with its method of eye tracking based on the light intensity data-eye rotation angle curve, can achieve efficient eye tracking with low requirements and cost and high accuracy.

[0085] According to an embodiment of the present invention, the process of deriving eye rotation angles based on light intensity data from light sensors can be implemented through simulation. As shown in FIG15 , this simulation process accurately generates a light intensity data-eye rotation angle curve for each light sensor. Specifically, a user can interact with an eye tracking device, which also includes a memory. For a current user, a processor determines whether a preset mapping relationship for the current user is stored in the memory; if so, the stored preset mapping relationship is retrieved to determine the eye rotation angle based on the retrieved preset mapping relationship; if not, a preset mapping relationship is generated for the current user.

[0086] Specifically, as shown in Figure 15, when a user wears a device including an eye tracking device, the device identifies the user's identity. If the user is an old user, it means that a preset mapping relationship (mapping function) for the current user has been stored before. The mapping function for the current user is directly loaded and then the data of n light sensors are collected to form n-dimensional vector data. The processor inputs the n-dimensional vector data into the mapping function, and the mapping function processes the vector and outputs the line of sight direction result, which includes the eye rotation angle.

[0087] If the current user is a new user, it means that no preset mapping relationship (mapping function) has been stored for the current user. In this case, interaction with the user is required to generate a preset mapping relationship (mapping function). For example, when indication information is displayed on a display screen and the user looks at the indication information, data from n light sensors is collected to form n-dimensional vector data (sample data). By obtaining n-dimensional vector data at multiple specific eye rotation angles and performing data fitting based on the n-dimensional vector data and multiple specific eye rotation angles, a mapping function is obtained and stored. The mapping function can also be verified to improve its correctness and its parameters can be optimized.

[0088] In another aspect of the present invention, the present invention provides an eye tracking method implemented using the eye tracking device described above. According to an embodiment of the present invention, the eye tracking method is applied to a processor, and the method includes: obtaining light intensity data based on light sensing data, wherein the light sensing data is obtained based on the light reflected by the eyeball, and / or based on the light reflected by the reflective layer; determining the eyeball rotation angle based on the light intensity data and a preset mapping relationship, wherein the preset mapping relationship represents the relationship between light intensity and eyeball rotation angle. Thus, the light emitted by the light source is incident on the eyeball, and the eyeball reflects at least part of the light to the reflective layer, and the reflective layer reflects the light reflected by the eyeball again and is received by the light sensor, and at the same time, the light sensor can further receive the light emitted by the light source that is directly reflected by the eyeball, and the light sensor obtains light sensing data based on the received reflected light. The processor obtains light intensity data based on the light sensing data, and can determine the eyeball rotation angle based on the light intensity data and the preset mapping relationship, thereby achieving eye tracking. At the same time, due to the provision of the reflective layer, the eye tracking device with the above structure can also overcome the problem of too small an exit pupil distance in eye tracking devices, thereby accurately tracking the eye position and expanding the field of view (FOV). Furthermore, by simultaneously capturing light intensity data corresponding to the light captured by each light sensor, a curve plotting the light intensity data captured by light sensor 700 and the rotation angle of eye 400 can be used to determine the eye angle corresponding to each light intensity data, thereby accurately achieving eye tracking.

[0089] According to an embodiment of the present invention, when the orthographic projections of multiple light sensors on a person's face do not overlap, a curve graph with poor correlation can be better obtained, which is more conducive to obtaining a closer eye rotation angle and performing eye tracking more accurately.

[0090] At the same moment, each light sensor captures its own light intensity data. Through the curve of the light intensity data of the light captured by the light sensor 700 and the rotation angle of the eyeball 400, the eyeball angle corresponding to each light intensity data can be found. At the same moment, the eyeball angle corresponding to each light sensor is the same. The more light sensors there are, the more eyeball angles corresponding to the same position at the same moment, and the more they can be used to prove that the eyeball at this moment is at this angle. It can be seen from this that the eye tracking device of the present invention, combined with its method for eye tracking based on the light intensity data-eyeball rotation angle curve, can efficiently achieve eye tracking, and has low requirements and costs and high accuracy. Among them, the requirements for obtaining the light intensity data of the light captured by the light sensor 700 and the curve of the eyeball 400 rotation angle are consistent with those described above and will not be elaborated on here.

[0091] According to an embodiment of the present invention, the process of the light sensor obtaining the eye rotation angle based on the received light intensity data can be realized through simulation, as shown in Figure 15. Through the simulation processing, the light intensity data-eye rotation angle curve of each sensor can be accurately obtained and the eye rotation angle can be determined in real time based on the generated curve.

[0092] In another aspect of the present invention, a head-mounted display device is provided. According to an embodiment of the present invention, the head-mounted display device includes the eye tracking device described above, and the head-mounted display device is also used to implement the eye tracking method. As a result, the head-mounted display device has a suitable exit pupil distance, accurately tracks the eye position, and has a large field of view, thereby effectively improving the interactive experience of the head-mounted display device. Those skilled in the art will understand that the head-mounted display device has all the features and advantages of the eye tracking device described above, and no further details will be given here.

[0093] In the embodiments of the present invention, there are no special requirements for the specific type of the head-mounted display device, and those skilled in the art can flexibly choose according to actual needs. For example, the head-mounted display device of the present invention can be an AR, VR or other head-mounted display device.

[0094] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0095] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An eye tracking device, characterized in that: include: frame; an optical display module, the optical display module being arranged on the frame; at least one light source, the light source being disposed on the frame, and the light emitted by the light source being reflectable by the eyeball; at least one reflective layer, the reflective layer being disposed on the optical display module and configured to reflect the light emitted by the light source reflected by the eyeball; at least one light sensor, the light sensor being disposed on the frame and configured to receive light reflected by the reflective layer and / or light from the light source reflected by the eyeball, obtain light sensing data, and transmit the light sensing data to a processor; A processor is provided on the frame, and is used to receive the light sensing data from the light sensor, obtain light intensity data based on the light sensing data, and determine the eye rotation angle based on the light intensity data and a preset mapping relationship, wherein the preset mapping relationship represents the relationship between light intensity and eye rotation angle.

2. The eye tracking device according to claim 1, wherein: In the case where the at least one light sensor includes multiple light sensors, the preset mapping relationship includes multiple relationship curves, and the multiple relationship curves correspond one-to-one to the multiple light sensors; the changing trends of the light intensity data in each relationship curve with the eye rotation angle are not completely consistent.

3. The eye tracking device according to any one of claims 1-2, characterized in that The optical display module includes: Display screen; The folding optical path lens is located on the side of the display screen close to the eyeball.

4. The eye tracking device according to claim 3, wherein: The setting position of the reflective layer meets at least one of the following conditions: The reflective layer is arranged on the surface of the display screen close to the folded light path lens; The reflective layer is arranged between the display screen and the folded light path lens; The reflective layer is arranged on the surface of the folded light path lens close to the display screen; The reflective layer is arranged on a surface of the folding light path lens away from the display screen.

5. The eye tracking device according to claim 1, wherein: The light sensor includes at least one of the following: a shooting camera, wherein the light sensing data includes image data, and the processor is configured to sum a plurality of pixel values ​​in the image data to obtain a pixel sum value, and use the pixel sum value as the light intensity data; The photoelectric detector comprises a light sensing data including a current signal, and the processor is configured to obtain the light intensity data according to the current signal.

6. The eye tracking device according to claim 1, wherein: The preset mapping relationship includes a mapping function, wherein the mapping function takes the eye rotation angle as an independent variable and the light intensity as a dependent variable; Determining the eyeball rotation angle based on the light intensity data and a preset mapping relationship includes inputting the light intensity data as a dependent variable into the mapping function and outputting the eyeball rotation angle.

7. The eye tracking device according to claim 1 or 6, characterized in that: The preset mapping relationship is generated in the following way: The processor generates indication information and displays it on the display screen, wherein the display position of the indication information can represent the eye movement angle of the user when looking at the indication information; When the user is gazing at the indication information, the user's eyeballs move, and light sensing data samples are obtained through the at least one light sensor; The processor determines an eye rotation angle sample based on the display position of the indication information, obtains a light intensity data sample based on the light sensing data sample, and fits and generates the preset mapping relationship based on the eye rotation angle sample and the light intensity data sample.

8. The eye tracking device according to claim 7, wherein: The processor is further configured to: For a current user, determining whether a preset mapping relationship for the current user is stored; If yes, retrieve a stored preset mapping relationship to determine the eye rotation angle based on the retrieved preset mapping relationship; If not, generate the preset mapping relationship for the current user.

9. The eye tracking device according to claim 1 or 5, characterized in that: The light sensor is arranged facing the reflective layer and / or the eyeball so as to receive light from the reflective layer and / or the eyeball to obtain the light sensing data.

10. The eye tracking device according to claim 3, wherein: The frame includes a connected frame body and a supporting component, the light source and the light sensor are arranged on the supporting component, and / or the light source and the light sensor are arranged on the frame and close to the edge of the folded light path lens.

11. The eye tracking device according to claim 10, wherein: The orthographic projections of the plurality of light sensors on the user's face do not overlap.

12. The eye tracking device according to claim 10, wherein: In the case where the at least one light source includes a plurality of light sources, the plurality of light sources are arranged in a plurality of rows and columns, and / or the plurality of light sources are dispersedly arranged around the edge of the folded light path lens.

13. The eye tracking device according to claim 12, wherein: The light sensor is arranged between the reflective layer and the eyeball and distributed on the edge of the folded light path lens, and / or the light sensor is arranged in the gap between the light sources.

14. The eye tracking device according to any one of claims 1-2, characterized in that The thickness of one layer of the reflective layer is 50-500 nm, and the visible light transmittance of one layer of the reflective layer is greater than or equal to 90%.

15. The eye tracking device according to any one of claims 1-2, characterized in that: The reflective layer includes a plurality of sub-reflective layers stacked together.

16. The eye tracking device according to any one of claims 1-2, characterized in that: The light source includes an infrared light source.

17. An eye tracking method implemented using the eye tracking device according to any one of claims 1 to 16, characterized in that: include: Obtaining light intensity data based on light sensing data, wherein the light sensing data is obtained based on light reflected by an eyeball and / or based on light reflected by a reflective layer; The eyeball rotation angle is determined based on the light intensity data and a preset mapping relationship, wherein the preset mapping relationship represents the relationship between the light intensity and the eyeball rotation angle.

18. A head-mounted display device, characterized in that: An eye tracking device comprising any one of claims 1-16 or used to implement the eye tracking method according to claim 17.

Images