Eye tracking device and eye tracking method

Abstract

An eye tracking device includes M light sources configured to emit N different types of light to a user's eye, M is a positive integer greater than or equal to 2, N is a positive integer greater than or equal to 2, and N is less than or equal to M; a camera module including a light filtering assembly and an image sensor; wherein the light filtering assembly is configured to filter at least part of the light reflected after the M light sources illuminate the eye, the light filtering assembly includes a plurality of light filtering units, and at least N sub-regions are included in the light filtering units, wherein the light filtering units filter at least the N different types of light, and one sub-region filters one type of light; the image sensor is configured to obtain the filtered at least part of the light to obtain an eye pupil image. The eye tracking device can accurately match the light spot and the light source.

Description

Technical Field

[0001] This application relates to the field of terminal and communication technology, and in particular to eye-tracking devices and eye-tracking methods. Background Technology

[0002] Electronic devices can use user actions as input to perform different operations or functions. For example, an electronic device can respond to a user's touch gestures to perform actions such as opening an application. Similarly, eye movements can be tracked and detected (i.e., eye tracking), allowing the electronic device to perform different operations or functions in response to the results of eye tracking. In head-mounted devices, eye tracking can be used to perform operations such as scrolling and opening applications; furthermore, eye tracking can be applied to enhance display effects. For instance, when a head-mounted device is displaying a display, eye tracking is applied to gaze-point rendering (i.e., the location where the user's eyes are looking), rendering high-quality images in the user's visual field's focal area (i.e., the area where the gaze point is located), providing the user with a better visual experience.

[0003] Eye tracking primarily uses the pupil-iris reflex method. A light source shines in front of the viewer's eyes; when this light reaches the eyes, it is reflected in the pupil or iris. A camera positioned in front of the viewer captures the reflected light and the image of the eye. The image formed by the reflected light on the camera is a light spot. Based on the light spot and the eye image, the viewer's gaze direction can be determined, thus revealing the gaze point. However, accurately determining the gaze point is a crucial problem that needs to be solved. Summary of the Invention

[0004] This application provides an eye-tracking device and an eye-tracking method that can accurately match light sources and light spots to precisely determine the gaze point of the human eye.

[0005] In a first aspect, embodiments of this application provide an eye-tracking device, the eye-tracking device comprising:

[0006] M light sources are configured to emit N different types of light to the user's eyes, with each light source emitting one type of light. M is a positive integer greater than or equal to 2, N is a positive integer greater than or equal to 2, and N is less than or equal to M.

[0007] The camera module includes a filter assembly and an image sensor; wherein,

[0008] The filter assembly is configured to filter at least a portion of the light reflected after the M light sources illuminate the eye. The filter assembly includes a plurality of filter units, each filter unit including at least N different types of sub-regions, wherein the different types of sub-regions filter different types of light.

[0009] The image sensor is configured to acquire at least a portion of the light passing through the filter assembly to obtain a human eye spot image, wherein the at least a portion of the light passing through the filter assembly forms at least one spot on the human eye spot image.

[0010] In the above embodiments, since each light source emits light of a corresponding type (e.g., wavelength), and each sub-region can transmit light of the corresponding type, when light from a certain light source reaches the filter component after being reflected by the human eye, if the light from that light source matches a certain sub-region, then the light from that light source can pass through that sub-region and form a light spot on the image sensor. Therefore, the light spot on the human eye's light spot image can reflect which type or types of light formed it, which helps improve the accuracy and efficiency of light spot matching and enhances the precision of the human eye's gaze point determined by eye tracking.

[0011] In some possible implementations, the image sensor includes an array of pixels, with each sub-region corresponding to a pixel. A light spot comprises multiple pixels, and each light spot corresponds to N different types of sub-regions. This correspondence between sub-regions and pixels ensures that the presence or absence of light transmission in each sub-region can be reflected in the human eye's light spot image. This allows for more precise determination of the type of light forming the light spot, improving the accuracy of eye tracking.

[0012] For example, when one or more of the following conditions are met, it is more advantageous to ensure that whether or not light passes through each sub-region can be represented in the human eye's light spot image:

[0013] The area of ​​this sub-region is less than or equal to 5 pixels of this image sensor;

[0014] The area of ​​this sub-region is less than or equal to 1 / N of the area of ​​one of these light spots; or,

[0015] This sub-region corresponds one-to-one with the pixels of the image sensor.

[0016] In some embodiments, M is greater than or equal to 4, and / or N is greater than or equal to 4. When M and N are greater than or equal to 4, there are more light sources and more types of light, which is more conducive to distinguishing different light sources and can more accurately match light sources and light spots.

[0017] In some embodiments, N is less than M, and at least two of the light sources emit the same type of light, which reduces the requirements for the types of light sources and filters. Two light sources emitting the same type of light are not adjacent, effectively avoiding interference between adjacent light sources. For example, the M light sources include multiple light source groups, each group containing at least two light sources emitting the same type of light, while light sources in different light source groups emit different types of light. This ensures effective identification and matching of light sources while reducing the requirements for light sources, filters, and other devices.

[0018] In some embodiments, two light sources emitting the same type of light may have different emission frequencies or emission times, which can further distinguish light sources emitting the same type of light. This ensures effective identification and matching of light sources while reducing the requirements for light sources, filters, and other devices.

[0019] Secondly, embodiments of this application provide an eye-tracking method, the method comprising:

[0020] Use M light sources to emit N different types of light towards the user's eyes, where M is a positive integer greater than or equal to 2, N is a positive integer greater than or equal to 2, and N is less than or equal to M;

[0021] The camera module has a filter component that filters at least a portion of the light reflected after the M light sources illuminate the eye; the filter component includes multiple filter units, each filter unit including at least N different types of sub-regions, wherein the different types of sub-regions filter different types of light;

[0022] The image sensor in the camera module acquires at least a portion of the light passing through the filter component to obtain a human eye spot image, wherein the at least a portion of the light passing through the filter component forms at least one spot on the human eye spot image.

[0023] In the above embodiments, since each light source emits light of a corresponding type (e.g., wavelength), and each sub-region can transmit light of the corresponding type, when the light from a certain light source reaches the filter component, if the light from that light source matches a certain sub-region, the light from that light source can pass through that sub-region and form a light spot on the image sensor. Therefore, the light spot on the human eye's light spot image can reflect which type or types of light formed it, which helps improve the accuracy and efficiency of light spot matching and enhances the precision of the human eye's gaze point determined by eye tracking.

[0024] In some possible implementations, the sub-region corresponds to the pixels of the image sensor, and one light spot includes multiple pixels, with one light spot corresponding to N different types of sub-regions. This correspondence between sub-regions and pixels ensures that whether or not light passes through each sub-region can be reflected in the human eye's light spot image, allowing for more precise determination of the type of light forming the light spot and improving the accuracy of eye tracking.

[0025] For example, when one or more of the following conditions are met, it is more advantageous to ensure that whether or not light passes through each sub-region can be represented in the human eye's light spot image:

[0026] The area of ​​this sub-region is less than or equal to 5 pixels of this image sensor;

[0027] The area of ​​this sub-region is less than or equal to 1 / N of the area of ​​one of these light spots; or;

[0028] This sub-region corresponds one-to-one with the pixels of the image sensor.

[0029] In some embodiments, M is greater than or equal to 4, and / or N is greater than or equal to 4. When M and N are greater than or equal to 4, there are more light sources and more types of light, which is more conducive to distinguishing different light sources and can more accurately match light sources and light spots.

[0030] In some embodiments, N is less than M, and at least two of the light sources emit the same type of light, which reduces the requirements for the types of light sources and filters. Two light sources emitting the same type of light are not adjacent, effectively avoiding interference between adjacent light sources. For example, the M light sources include multiple light source groups, each group containing at least two light sources emitting the same type of light, while light sources in different light source groups emit different types of light. This ensures effective identification and matching of light sources while reducing the requirements for light sources, filters, and other devices.

[0031] In some embodiments, two light sources emitting the same type of light may have different emission frequencies or emission times, which can further distinguish light sources emitting the same type of light. This ensures effective identification and matching of light sources while reducing the requirements for light sources, filters, and other devices.

[0032] Thirdly, embodiments of this application also provide an electronic device, characterized in that the electronic device includes one or more processors, M light sources, a filter assembly, an image sensor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions, the instructions being used for:

[0033] The gaze direction of the human eye is obtained based on the human eye spot image.

[0034] Fourthly, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a computer, obtain the gaze direction of a human eye based on the human eye spot image.

[0035] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, obtain the gaze direction of the human eye based on the human eye spot image. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a VR system according to an embodiment of this application;

[0037] Figure 2A This is a schematic diagram of a VR head-mounted display device provided in an embodiment of this application;

[0038] Figure 2B This is a simplified schematic diagram of the VR head-mounted display device provided in the embodiments of this application;

[0039] Figure 3 This is an exemplary optical display module provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the structure of a VR head-mounted display device provided in an embodiment of this application;

[0041] Figure 5 This is an exemplary software structure block diagram of a VR head-mounted display device provided in an embodiment of this application;

[0042] Figure 6 This is a cross-sectional schematic diagram of an eyeball structure model in an embodiment of this application;

[0043] Figure 7 This is an exemplary schematic diagram of an eye-tracking module in a VR head-mounted display device that uses the pupil-corneal reflection method to achieve eye tracking;

[0044] Figure 8 This is an exemplary schematic diagram of a human eye light spot image in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of a scene in which a light source illuminates the eye to obtain a light spot image of the human eye in an embodiment of this application;

[0046] Figure 10 This is an exemplary schematic diagram of the filter on the sensor of the light source and camera module in the eye-tracking module of this application embodiment;

[0047] Figure 11 This is another exemplary schematic diagram of the human eye spot image obtained in the embodiments of this application;

[0048] Figure 12 This is an exemplary schematic diagram of eight light sources arranged in several regular shapes in an embodiment of this application;

[0049] Figure 13 This is an exemplary schematic diagram of eight light sources arranged in several irregular shapes in an embodiment of this application;

[0050] Figure 14 These are schematic diagrams illustrating several exemplary arrangement structures of the eight light sources in embodiments of this application;

[0051] Figure 15 This is an exemplary schematic diagram of the camera module's placement location in an embodiment of this application. Detailed Implementation

[0052] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0053] (1) The at least one involved in the embodiments of this application includes one or more; wherein, multiple means two or more. Furthermore, it should be understood that in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order. For example, the first region and the second region do not represent the degree of importance of the two, or their order, but are merely for distinguishing regions. In the embodiments of this application, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0054] (2) Virtual Reality (VR) technology is a human-computer interaction method created using computer and sensor technologies. VR technology integrates various scientific technologies such as computer graphics, computer simulation, sensor technology, and display technology to create virtual environments. These virtual environments include computer-generated, real-time dynamic, three-dimensional, realistic images that provide visual perception to the user. In addition to the visual perception generated by computer graphics, there are also auditory, tactile, force, and motion perceptions, and even olfactory and gustatory perceptions, also known as multi-sensory perception. Furthermore, it can detect the user's head movements, eye movements, gestures, or other human actions. The computer processes the data corresponding to the user's actions and responds to them in real time, feeding back to the user's five senses, thus forming the virtual environment. For example, when wearing VR wearable devices, users can see a VR game interface and interact with it through gestures, controllers, etc., as if they were actually in the game.

[0055] (3) Augmented Reality (AR) technology refers to superimposing computer-generated virtual objects onto real-world scenes to enhance the real world. In other words, AR technology requires capturing real-world scenes and then adding virtual environments to the real world.

[0056] Therefore, the difference between VR and AR technologies lies in the fact that VR technology creates a completely virtual environment where users see only virtual objects; while AR technology overlays virtual objects onto the real world, including both real and virtual objects. For example, a user wearing transparent glasses can see the surrounding real environment through these glasses, and virtual objects can also be displayed on the glasses, allowing the user to see both real and virtual objects.

[0057] (4) Mixed Reality (MR) technology enhances the realism of the user experience by introducing real-world scene information (or real-world scene information) into a virtual environment, thus building an interactive feedback bridge between the virtual environment, the real world, and the user. Specifically, real-world objects are virtualized (for example, using a camera to scan real-world objects for 3D reconstruction to generate virtual objects), and the virtualized real objects are introduced into the virtual environment, so that users can see real objects in the virtual environment.

[0058] It should be noted that the technical solutions provided in this application can be applied to scenarios where the electronic devices used for eye tracking are head-mounted devices, such as VR, AR, or MR scenarios. They can also be applied to other scenarios where the electronic devices used for eye tracking are not head-mounted devices, such as scenarios where eye tracking is performed using terminal devices (e.g., mobile phones, tablets, etc.), computer monitors, smart cars, or large-screen devices such as televisions. For example, in the driving scenario of a smart car, the technical solutions provided in this application can more accurately determine the gaze point of the human eye and perform eye tracking more quickly and accurately. In short, it is applicable to any scenario that requires accurate determination of the gaze point of the human eye for eye tracking.

[0059] For ease of understanding, the following text will mainly use VR scenarios as an example.

[0060] For example, please see Figure 1 This is a schematic diagram of a VR system according to an embodiment of this application. The VR system includes a VR wearable device (in this embodiment, a VR head-mounted display device 100 is used as an example) and an image processing device 200. This VR system can be called a VR split-type device. The VR head-mounted display device 100 can be connected to the processing device 200. The connection between the VR head-mounted display device 100 and the processing device 200 includes wired or wireless connections. The wireless connection can be Bluetooth (BT), traditional Bluetooth or Bluetooth Low Energy (BLE) Bluetooth, wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Zigbee, frequency modulation (FM), near field communication (NFC), infrared (IR), or general 2.4G / 5G frequency band wireless communication connections, etc.

[0061] In some embodiments, the image processing device 200 can perform processing calculations. For example, the image processing device 200 can generate an image and process the image (the processing method will be described later), and then send the processed image to the VR head-mounted display device for display. The image processing device 200 may include a host (e.g., a VR host) or a server (e.g., a VR server). The VR host or VR server can be a device with significant computing power. For example, the VR host can be a mobile phone, tablet, laptop, or other device, and the VR server can be a cloud server, etc.

[0062] In some embodiments, the VR head-mounted display device 100 can be glasses, a helmet, etc. The VR head-mounted display device 100 generally has two display devices, namely display device 110 and display device 120. The display devices of the VR head-mounted display device 100 can display images to the human eye. Figure 1 In the illustrated embodiment, display devices 110 and 120 are enclosed inside the VR glasses, so Figure 1 The arrows used to indicate display device 110 and display device 120 are represented by dashed lines.

[0063] In some embodiments, the VR head-mounted display device 100 also has image generation and processing functions, that is, the VR head-mounted display device 100 does not require Figure 1 The image processing device 200 in the VR head-mounted display device 100 can be called a VR all-in-one machine.

[0064] Figure 2A This is a schematic diagram of a VR head-mounted display device 100. Figure 2A As shown in (a), the VR head-mounted display device 100 includes an optical display module 210 and an optical display module 220. The optical display module 210 includes a display device and an optical device 211. The optical display module 220 includes a display screen and an optical device 221. In some embodiments, the optical display modules 210 and 220 further include optical devices. In some embodiments, the optical display modules 210 and 220 can each be two hollow cylindrical lenses, with the optical devices housed within the lenses. The optical devices and display devices are mounted on the VR head-mounted display device 100 via the lenses. When a user wears the VR head-mounted display device 100, the optical display module 210 can be used to display an image to the user's left eye. The optical display module 220 can be used to display an image to the user's right eye. It is understood that... Figure 2A The VR headset 100 shown in (a) may also include other components, such as a support 230 and a bracket 240. The support 230 supports the VR headset 100 on the bridge of the nose, and the bracket 240 supports the VR headset 100 on the ears to ensure stable wearing of the VR headset 100. Figure 2AAs shown in (b), at least one eye-tracking module (including M light sources 2501 and at least one camera module 2502) can be provided on the VR head-mounted display device 100 to track the movement of the human eye and thereby determine the gaze point of the human eye. In some embodiments, the end face 100a of the VR head-mounted display device 100 facing the face (or the user's eyes) can be provided with light sources 2501. In addition, the end face 100a of the VR head-mounted display device 100 facing the face (or the user's eyes) can also be provided with camera modules 2502. For example, the light source 2501 is located on the end face 210a of the lens barrel facing the eye, and the end face 210a of the lens barrel facing the eye can be understood as a part of the end face 100a. Taking the eye-tracking module 250 on the optical display module 210 as an example, in some embodiments, the eye-tracking module 250 includes light sources 2501 and camera modules 2502. For example, eight light sources 2501 are arranged on the eye-facing end face 210a of the lens barrel. The eight light sources 2501 can be evenly distributed in a circle, and the camera module 2502 can be disposed on the eye-facing end face 210a of the lens barrel. The light sources 2501 and the camera module 2502 can both be arranged around the eye-facing side of the optical device 211.

[0065] For ease of description, please refer to Figure 2B , Figure 2B This can be understood as... Figure 2A A simplified version of the VR head-mounted display device 100, for example, Figure 2B Only optical display modules 210 and 220 are shown; other components are not shown. Figure 2BWhen a user wears the VR headset 100, the display device 110 is located on the side of the optical device 211 away from the left eye, and the display device 120 is located on the side of the optical device 221 away from the right eye. The optical devices 211 and 221 are symmetrical with respect to the center line of the face or the center line D of the VR headset 100. The center line of the face can be the perpendicular bisector between the right and right eyes; the center line D of the VR headset 100 can be the center line of the end face 100a, or the center line of the support 240, etc. When the display device 110 displays an image, the light emitted by the display device 110 converges to the user's left eye through the optical device 211. When the display device 120 displays an image, the light emitted by the display device 120 converges to the user's right eye through the optical device 221. In some embodiments, the VR head-mounted display device 100 may further include optical elements 212 and 222, with optical elements 212 and 211 forming a lens assembly, and optical elements 222 and 221 forming another lens assembly. A lens assembly may include at least one optical element, and one or more optical elements in the lens assembly may be adjustable to change the optical power of the lens assembly. For example, the position of one or more optical elements in the lens assembly may be moved away from or closer to the display device to change the optical power.

[0066] It should be noted that, Figure 2A or Figure 2B The VR headset display device 100 shown is only a logical illustration. In specific implementations, the number of optical devices and / or display devices can be flexibly set according to different needs. For example, in some embodiments, display device 110 and display device 120 can be two independent display devices, or two display areas on the same display device. In some embodiments, display device 110 and display device 120 can be displays, such as liquid crystal displays, light-emitting diode (LED) displays, or other types of display devices, which are not limited in this application embodiment. In other embodiments, optical device 212 and optical device 211 can be two independent optical devices, or different parts of the same optical device. In some embodiments, optical device 212 and optical device 211 can be one or more optical devices such as reflectors, transmissive mirrors, or optical waveguides, which can also improve the field of view. For example, optical device 212 and optical device 211 can be lens groups composed of multiple transmissive mirrors. For example, optical devices can be Fresnel lenses and / or aspherical lenses, which are not limited in this application embodiment.

[0067] Please see Figure 3This is an exemplary optical display module 210. The display device 110 in this optical display module 210 is a display screen 110. The optical display module 210 includes five optical components (such as 301-305), forming a pancake folding optical lens group (hereinafter referred to as the pancake lens group). The following description, taking light emitted from the display screen 110 of the optical display module 210 and passing through the pancake lens group into the human eye as an example, explains the structure of the pancake lens group and its light processing mechanism during operation:

[0068] like Figure 3 As shown, the Pancake lens assembly may include at least five optical components (such as 301-305). Each of the optical components 301-305 may be implemented as an optical lens with a corresponding function, or by achieving the corresponding function of the component through optical coating on adjacent lenses or other optical parts.

[0069] In this example, 301 can be a polarizer (P). 302 can be a quarter-wave plate (QWP). 303 can be implemented using a beam splitter (BS). 304 can be a quarter-wave plate. 305 can be implemented using a polarization reflector (PR).

[0070] like Figure 3 As shown, after incident at 301, the incident light can be transmitted sequentially in the order of 302-303-304-305.

[0071] For example, light can be modulated into linearly polarized light (i.e., plane-polarized light) after passing through 301. In some embodiments, the modulation direction of 301 can be set to the y-axis direction. Thus, the incident light after passing through 301 can be modulated into linearly polarized light in the y-axis direction. Then, this linearly polarized light can pass through 302, thereby being adjusted into rotationally polarized light. For example, with the fast axis of 302 at 45° to the y-axis, the linearly polarized light can be adjusted into right-handedly polarized light after passing through 302, wherein the waveplate has a fast axis and a slow axis, which are perpendicular to each other, and the polarized light has a slightly higher speed in the fast axis direction. This right-handedly polarized light can be incident on 303. Due to the semi-transparent and semi-reflective properties of 303, a portion of the right-handedly polarized light can be transmitted through 303, while another portion is reflected by 303. The right-handedly polarized light transmitted through 303 can be incident on 304. With the fast axis direction of 304 the same as that of 302, the right-handed polarized light transmitted through 303 can directly penetrate 304 and be incident on 305. The right-handed polarized light incident on 305 can be modulated into linearly polarized light along the x-axis direction and reflected at the surface of 305.

[0072] Light reflected by 305 can pass through 304-303 and is reflected at 303.

[0073] For example, light reflected from surface 305 can pass through 304 and be modulated into right-handed polarized light, a portion of which can be reflected from surface 303. It should be noted that after reflection from surface 303, this right-handed polarized light can be modulated into left-handed polarized light.

[0074] Left-handed polarized light reflected by 303 can exit the Pancake lens group through 304-305 and eventually enter the human eye.

[0075] For example, after passing through 304, the left-handed polarized light can be modulated into linearly polarized light (i.e., plane-polarized light) with its polarization direction along the y-axis. Then, this linearly polarized light along the y-axis can exit the pancake lens assembly through 305 and enter the human eye. It is understood that in some embodiments, the polarization transmission characteristics of 305 can be set to transmit linearly polarized light along the y-axis, thereby ensuring that the linearly polarized light along the y-axis can exit smoothly from 305.

[0076] In this way, light can be folded and transmitted within the Pancake lens group in the sequence 301-302-303-304-305-304-303-304-305, thus achieving the effect of folding the light path. Therefore, it is possible to achieve the transmission of a longer light path within a relatively small space (such as inside the optical display module 210 of a VR wearable device).

[0077] exist Figure 2AIn the example shown in (b), the camera module 2502 can be located on the end face of the optical display module 210 near the face, and can be located between any two light sources 2501. For example, the camera module 2502 can be located between the two light sources 2501 at the lower right. In this way, the light reflected from the eyes by the light sources 2501 does not need to be refracted by the lens group and can be directly received by the camera module 2502, which can make the eye tracking calculation more accurate.

[0078] In some embodiments, the camera module 2502 may be located on the side of one of the optics facing away from the user's eye, so that the camera module 2502 can be further away from the user's eye to receive more light reflected from the eye, thereby obtaining more information about the light spot and improving detection accuracy. For example, in Figure 3 In this configuration, the camera module 2502 can be located between 303 and 302, or between other adjacent optical components. It is understood that although the camera module 2502 is located between the optical components, it is outside the range of the optical path transmission to avoid blocking the light from the display screen. In some embodiments, the camera module 2502 can be located between the lens group and the display screen 110. Again, it is understood that although the camera module 2502 is located between the lens group and the display screen 110, it is outside the range of the optical path transmission to avoid blocking the light from the display screen. This allows the camera module 2502 to be positioned optimally according to the reflected light from the eye illuminated by the receiving light source, making the focus of attention acquired by eye tracking more accurate.

[0079] It is understandable that the VR head-mounted display device 100 may include more components. For example, please refer to... Figure 4 This illustration shows a structural schematic diagram of a VR head-mounted display device 100 provided in an embodiment of this application. Figure 4 As shown, the VR head-mounted display device 100 may include a processor 401, a memory 402, a sensor module 403 (which can be used to acquire the user's posture), a microphone 404, a button 405, an input / output interface 406, a communication module 407, a camera 408, a battery 409, an optical display module 410, and an eye-tracking module 412, etc.

[0080] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the VR head-mounted display device 100. In other embodiments of this application, the VR head-mounted display device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0081] Processor 401 is typically used to control the overall operation of the VR head-mounted display device 100 and may include one or more processing units. For example, processor 401 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The processor 401 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 401 is a cache memory. This memory can store instructions or data that the processor 401 has just used or that are used repeatedly. If the processor 401 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 401, and thus improves the efficiency of the system.

[0083] In some embodiments of this application, the processor 401 can acquire the human eye spot image sent by the camera module in the eye-tracking module 412, and can also know the position of the user's eyes, and then calculate the user's gaze point.

[0084] In some embodiments, the processor 401 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0085] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 401 may include multiple I2C buses.

[0086] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 401 and the communication module 407. For example, the processor 401 communicates with the Bluetooth module in the communication module 407 via the UART interface to implement Bluetooth functionality.

[0087] The MIPI interface can be used to connect the processor 401 to peripheral devices such as the display screen and camera 408 in the optical display module 410.

[0088] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 401 to the camera 408, the display screen in the optical display module 410, the communication module 407, the sensor module 403, the microphone 404, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 408 can capture images including real objects, and the processor 401 can fuse the images captured by the camera with virtual objects, displaying the fused image through the optical display module 410.

[0089] The USB interface conforms to the USB standard specification and can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the VR headset 100, and can also be used for data transfer between the VR headset 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB 3.0, used for compatibility with high-speed display port (DP) signal transmission, enabling the transmission of high-speed audio and video data.

[0090] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the VR head-mounted display device 100. In other embodiments of this application, the VR head-mounted display device 100 may also adopt different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0091] Additionally, the VR headset 100 may include wireless communication functionality; for example, the VR headset 100 can receive images from other electronic devices (such as a VR host) for display. The communication module 407 may include a wireless communication module and a mobile communication module. The wireless communication functionality can be implemented using an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR headset 100 may include multiple antennas, each of which can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, an antenna can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0092] The mobile communication module can provide wireless communication solutions for VR head-mounted display devices 100, including 2G, 3G, 4G, and 5G networks. The mobile communication module may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves via an antenna, filter and amplify the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor and radiate it as electromagnetic waves via the antenna. In some embodiments, at least some functional modules of the mobile communication module may be housed in the processor 401. In some embodiments, at least some functional modules of the mobile communication module and at least some modules of the processor 401 may be housed in the same device.

[0093] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker), or displays images or videos through the display screen in the optical display module 410. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 401 and may be housed in the same device as the mobile communication module or other functional modules.

[0094] The wireless communication module can provide solutions for wireless communication applications in the VR head-mounted display device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 401. The wireless communication module can also receive signals to be transmitted from the processor 401, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0095] In some embodiments, the antenna of the VR headset 100 is coupled to the mobile communication module, enabling the VR headset 100 to communicate with networks and other devices via wireless communication technologies. These wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0096] The VR headset 100 implements display functions through a GPU, an optical display module 410, and an application processor. The GPU is a microprocessor for image processing, connected to the optical display module 410 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 401 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0097] The memory 402 can be used to store computer executable program code, which includes instructions. The processor 401 executes various functional applications and data processing of the VR headset 100 by running the instructions stored in the memory 402. The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the VR headset 100 (such as audio data, phone book, etc.). In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0098] The VR headset 100 can implement audio functions through an audio module, speaker, microphone 404, headphone jack, and application processor. Examples include music playback and recording. The audio module converts digital audio information into analog audio signals for output, and also converts analog audio input into digital audio signals. The audio module can also encode and decode audio signals. In some embodiments, the audio module can be located in the processor 401, or some functional modules of the audio module can be located in the processor 401. The speaker, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. The VR headset 100 can listen to music or make hands-free calls through the speaker.

[0099] Microphone 404, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The VR headset 100 may be equipped with at least one microphone 404. In some embodiments, the VR headset 100 may be equipped with two microphones 404, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the VR headset 100 may also be equipped with three, four, or more microphones 404, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0100] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface, or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface, or a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0101] In some embodiments, the VR headset 100 may include one or more buttons 405 that can control the VR wearable device, providing users with access to functions on the VR headset 100. The buttons 405 may take the form of buttons, switches, dials, and touch or proximity sensing devices (such as touch sensors). Specifically, for example, a user can press a button to turn on the optical display module 410 of the VR headset 100. Buttons 405 may include a power button, volume buttons, etc. Buttons 405 may be mechanical buttons or touch buttons. The VR headset 100 can receive button inputs and generate key signal inputs related to user settings and function control of the VR headset 100.

[0102] In some embodiments, the VR headset 100 may include an input / output interface 406, which can connect other devices to the VR headset 100 via suitable components. Components may include, for example, audio / video jacks, data connectors, etc.

[0103] The optical display module 410 is used to present images to the user under the control of the processor 401. The optical display module 410 can use one or more optical devices such as mirrors, transmissive mirrors, or optical waveguides to convert real-pixel image display into near-eye projection of virtual image display, realizing a virtual interactive experience or an interactive experience combining virtual and reality. For example, the optical display module 410 receives image data information sent by the processor 401 and presents the corresponding image to the user. In some embodiments, the optical display module 410 may include optical display module 210 and optical display module 220.

[0104] In embodiments of this application, the VR head-mounted display device 100 further includes an eye-tracking module 412. The eye-tracking module 412 is used to track the movement of the human eye, thereby determining the gaze point. For example, image processing technology can be used to locate the pupil position, obtain the pupil center coordinates, and then calculate the person's gaze point. In some embodiments, the eye-tracking system can determine the user's gaze point position (or determine the user's gaze direction) using methods such as video eye diagrams, photodiode response methods, or pupil-corneal reflection methods, thereby achieving eye tracking.

[0105] It should be noted that in some embodiments of this specification, separate eye-tracking modules can be set for each of the user's eyes to perform eye tracking synchronously or asynchronously. In other embodiments of this specification, an eye-tracking module can be set only near one of the user's eyes. The eye-tracking module obtains the gaze direction of the corresponding eye, and based on the relationship between the fixation points of the two eyes (e.g., when a user observes an object through both eyes, the fixation points of the two eyes are generally close or the same), combined with the user's interocular distance, the gaze direction or fixation point position of the user's other eye can be determined.

[0106] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the VR head-mounted display device 100. In other embodiments of this application, the VR head-mounted display device 100 may include more than Figure 2A The embodiments of this application do not limit the number of components, the combination of certain components, the separation of certain components, or the different arrangement of components.

[0107] It is understood that the VR head-mounted display device 100 is an example of an electronic device in the embodiments of this application. The electronic device in the embodiments of this application can also take many other forms, such as AR wearable devices, MR wearable devices, in-vehicle eye-tracking display devices, smart mobile devices, large-screen displays, smart cars, computer monitors, etc., which are not limited here.

[0108] Figure 5 This is an exemplary software structure block diagram of a VR head-mounted display device 100 according to an embodiment of this application.

[0109] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: application layer 501, application framework layer 502, runtime 503, system library 504, and kernel layer 505.

[0110] Application layer 501 may include a series of application packages.

[0111] like Figure 5 As shown, the application package may include applications (also referred to as apps) such as camera 501A, calendar 501B, map 501C, WLAN 501D, music 501E, SMS 501F, gallery 501G, call 501H, navigation 501I, Bluetooth 501J, and video 501K.

[0112] The application framework layer 502 provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0113] like Figure 5 As shown, the application framework layer 502 may include a window manager 5021, a content provider 5022, a phone manager 5023, a resource manager 5024, a notification manager 5025, a view system 5026, etc.

[0114] Window Manager 5021 is used to manage window programs. Window Manager 5021 can obtain the screen size, determine if a status bar is present, lock the screen, capture the screen, etc.

[0115] Content provider 5022 is used to store and retrieve data, and to make that data accessible to applications. The data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0116] The phone manager 5023 is used to provide communication functions for the VR head-mounted display device 100. This includes, for example, managing call status (including connection and disconnection).

[0117] File Explorer 5024 provides applications with a variety of resources, such as localized strings, icons, images, layout files, video files, and more.

[0118] Notification Manager 5025 allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, Notification Manager can be used to notify users of download completion or message alerts. Notification Manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0119] The view system 5026 includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images.

[0120] The runtime (503) includes the core libraries and the virtual machine. The runtime is responsible for system scheduling and management.

[0121] The core library consists of two parts: one part is the functionalities that the programming language (e.g., Java) needs to call, and the other part is the system's core library.

[0122] Application layer 501 and application framework layer 502 run in a virtual machine. The virtual machine executes the programming files (e.g., .jave files) of application layer 501 and application framework layer 502 as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0123] System library 504 may include multiple functional modules. For example: surface manager 5041, 3D graphics processing library 5042 (e.g., OpenGL ES), 2D graphics engine 5043 (e.g., SGL), media library 5044, etc.

[0124] The Surface Manager 5041 is used to manage the display subsystem and provides fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0125] Media Library 5044 supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0126] The 5042 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0127] 2D graphics engine 5043 is a 2D graphics engine.

[0128] Kernel layer 505 is the layer between hardware and software. Kernel layer 505 includes at least display driver 5051, camera driver 5052, audio driver 5053, and sensor driver 5054.

[0129] In some embodiments of this application, the application framework layer 502 may further include an eye-tracking function module 5027, which is used to match the light spot in the human eye image acquired by the camera driver 5052 with the light source, calculate the user's gaze direction, and thus determine the user's gaze point. In other embodiments of this application, the eye-tracking function module 5027 may also be located in the application layer 501, the system library 504, or the kernel layer 505, which is not limited here.

[0130] like Figure 6A cross-sectional schematic diagram of an eyeball structure model provided in an embodiment of this application, as shown below. Figure 6 As shown, the eyeball includes the cornea 601, iris 602, pupil 603, sclera 604, lens 605, and retina 606, wherein:

[0131] The cornea 601 has a high reflectivity to light, thus forming a clear reflection point after illumination. The cornea 601 is the transparent part at the front of the eyeball, acting as the first barrier for light entering the eye. The central 3mm of the outer surface of the cornea 601 is a spherical arc surface, called the optical zone, which gradually increases in radius of curvature around the periphery, forming an aspherical shape. Figure 6 In the eyeball structure model shown, the cornea 601 is assumed to be a spherical arc surface.

[0132] The iris 602, which is covered by the cornea 601, is a disc-shaped membrane. The area it occupies is dark and roughly circular, with a central opening called the pupil 603. If the light is too strong, the sphincter muscle within the iris 602 contracts, causing the pupil 603 to constrict; if the light weakens, the dilator muscle of the iris 602 contracts, causing the pupil 603 to dilate. The pupil 603 is a small, round opening in the center of the iris in an animal or human eye, serving as the passageway for light to enter the eye. The lens 605 is a biconvex transparent tissue located behind the iris 602. Its shape and function are similar to a convex lens, clearly reflecting the images of objects at varying distances onto the retina 606. The retina 606 is the light-sensitive part of the eye, and images of external objects are formed on it.

[0133] The sclera 604 (also known as the white of the eye) has a low reflectivity to light. It is a major component of the eyeball wall, located at the junction with the cornea, and has a tough structure that supports and protects the intraocular tissues.

[0134] like Figure 7 This is an exemplary schematic diagram illustrating eye tracking using the pupillary corneal reflection method in an eye-tracking module 412 of a VR head-mounted display device 100. The eye-tracking module 412 may include one or more light sources 4121 and one or more camera modules 4122. Figure 7 The light source 4121 is illustrated using a near-infrared light-emitting diode (LED) as an example, and the camera module 4122 is illustrated using a near-infrared camera 4122 as an example. When the light emitted by the light source is invisible light (such as infrared or near-infrared light), the influence of the light source on the eyes can be avoided, and imperceptible eye tracking can be performed.

[0135] In some embodiments, the eye-tracking module 412 can obtain the user's gaze direction by: Near-infrared LED 1201 illuminating the human eye 701; the light from the near-infrared LED 4121 illuminating the human eye and being reflected in areas such as the cornea 601 or iris 602; and the near-infrared camera 4122 acquiring the reflected light and the human eye image to obtain a human eye spot image. In some embodiments, the near-infrared camera 4122 can acquire an image 702 of the reflected light (e.g., an image of the spot) and an image of the human eye (the human eye image includes an image 703 at the center of the pupil). The human eye spot image includes the image 702 of the reflected light (e.g., an image of the spot) and the human eye image. Based on the human eye spot image, the optical axis direction 704 of the eyeball can be determined, thereby obtaining the user's gaze direction.

[0136] In some embodiments, the process of obtaining the user's gaze direction can be divided into the following stages:

[0137] (1) Calibration of human eye parameters:

[0138] Because each user's eyeball is different, the eye parameters tracked and used in the pupillary corneal reflex method calculation process will also differ. To improve the accuracy of the final calculation results, the eye parameters are calibrated before using the pupillary corneal reflex method algorithm for subsequent algorithmic calculations.

[0139] The results of human eye parameter calibration include the angle between the user's visual axis and optical axis. The visual axis is the user's gaze direction, and the optical axis is the direction from the center of the user's pupil to the center of the cornea.

[0140] (2) Detection of light spots and pupils on the image:

[0141] After the human eye is illuminated by the light source 4121 in the eye-tracking module 412, the human eye will reflect part of the light illuminating the light source 4121 to the camera module 4122 in the eye-tracking module 412, and the camera module 4122 can obtain a light spot image of a human eye.

[0142] like Figure 8The image shown is an exemplary schematic diagram of a human eye light spot image. This human eye light spot image may include: light spot 801, which is the image of the reflection point of the camera module 4122 after the light source 4121 illuminates the cornea 601 region of the human eye; iris image 802, formed by the scattered light from the light source 4121 illuminating the iris 602 region of the human eye and then reaching the camera module 4122; sclera image 803, formed by the scattered light from the light source 4121 illuminating the sclera 604 region of the human eye and then reaching the camera module 4122; and human eye contour image 804, formed by the scattered light from the light source 4121 illuminating the human eye contour and then reaching the camera module 4122. The human eye light spot image may also include images of the pupil 603 center (not shown in the figure), eyelashes, eyelids, etc., located at the center of the iris 602 region.

[0143] exist Figure 8 In the human eye image, spot 801 is located in the iris image 802 region because, in the structure of the human eye, the cornea 601 has a high reflectivity, and the transparent cornea 601 covers the iris 602. Therefore, the light spots formed by the light reflected by the cornea 601 or the iris 602 are all located in the iris image 802 region. Furthermore, because the sclera 604 has a low reflectivity, the light spots formed by the light reflected by the sclera 604 are difficult for the camera module 4122 to capture.

[0144] After obtaining the human eye light spot image, the VR head-mounted display device 100 will perform high-precision detection on the human eye light spot image to determine the position of the light spot and the center of the pupil in the human eye light spot image.

[0145] (3) Matching of light spot source:

[0146] After detecting the location of the light spot and the center of the pupil in the human eye light spot image, to perform subsequent calculations using the pupil-corneal reflection method, it is necessary to match the light spot detected in the human eye light spot image with the light source in the three-dimensional physical world, that is, to find out which light source emitted the light that formed the light spot in the human eye light spot image.

[0147] (4) Calculate the positions of the membrane center and pupil center in three-dimensional physical space to determine the direction of the optical axis;

[0148] After the VR head-mounted display device 100 matches the light spot in the human eye light spot image with the light source 4121 of the eye-tracking module 412 in the three-dimensional physical space, it can use subsequent algorithms to calculate the position of the user's corneal center and pupil center in the three-dimensional physical space. The line connecting the corneal center and pupil center is taken as the optical axis of the user's eye when the human eye light spot image was acquired, thus calculating the direction of the optical axis.

[0149] (5) Based on human eye parameters and optical axis direction, the visual axis direction is obtained, which is the user's line of sight direction.

[0150] Based on the optical axis direction of the user's eye obtained in (4), and the angle between the optical axis direction and the visual axis direction of the user's eye obtained in (1), the visual axis direction of the user's eye can be calculated, which is the gaze direction of the user's eye. The user's gaze point on the display screen can also be further calculated based on the gaze direction of the user's eye.

[0151] Combination Figure 2A The VR head-mounted display device 100 shown is... Figure 9 This is a schematic diagram of a scene where light source 4121 illuminates the eye, resulting in a light spot image of the human eye. Figure 9 The arrangement and position of the light source 4121 and the camera module 4122 can be as follows: Figure 2A As shown. Figure 9 As shown, the case of a single eye will be explained separately. Figure 9 The coordinate system shown in (a) is a Cartesian coordinate system established with the center point of the plane containing the eight light sources 4121 (e.g., plane 210a). This Cartesian coordinate system is only used to calibrate the position of each device for ease of description. In practical applications, the VR head-mounted display device 100 may or may not establish a coordinate system; this is not a limitation here. The camera module 4122 is located to the lower right of this coordinate system. During the physical process of the light emitted from the eight light sources 4121 illuminating the eye and forming an image in the camera module 4122, the light also undergoes refraction through a series of lenses within the lens of the camera module 4122. In some embodiments, the number of light spots is less than or equal to the number of light sources. Figure 9 As shown in (b), the final human eye spot image can include the human eye outline 901, the iris area 902, and four spots 903.

[0152] In this situation, determining which of the eight light sources 4121 specifically generated the four light spots 903 through light reflection is quite difficult. If the light spots cannot be accurately matched with the light sources, the gaze point of the human eye cannot be accurately determined.

[0153] by Figure 9 Taking (b) as an example, the number of light spots 903 is less than the number of light sources. The reasons may include one or more of the following:

[0154] 1. Due to the rotation of the human eyeball, the position of the reflected point of a light source at a known location in space varies. (As shown above) Figure 6 As mentioned above, since the cornea 601 has a high reflectivity and the sclera 604 has a low reflectivity, when the reflection point of the light from the light source 4121 shining on the eye is on the sclera 604, the camera module 4122 may not be able to capture the light spot, which will result in the number of light spots in the obtained image being less than the actual number of light sources.

[0155] 2. When the human eye rotates to a certain angle, the light emitted by the light source 4121 at different positions undergoes different reflections or refractions, which may cause the reflection points of two or more light sources 4121 to coincide during the human eye reflection process. In other words, one light spot 903 on the human eye light spot image obtained by the camera module 4122 may correspond to multiple light sources.

[0156] 3. In order not to obstruct the lens assembly, the camera module 4122 is offset, that is, the camera module 4122 is not in the center of the multiple light sources 4121, and the light from the light source 4121 reaches the image sensor of the camera module 4122 after multiple refractions, resulting in a complex optical path.

[0157] Therefore, ambiguity will occur when matching light spots in this situation. That is, one light spot may correspond to multiple light sources and not all light sources have corresponding light spots. It is impossible to accurately distinguish which light source(s) each light spot corresponds to.

[0158] The eye-tracking module 412 provided in this application embodiment can match the light source and the light spot more quickly and accurately, eliminating ambiguity.

[0159] To simplify the description and facilitate understanding, the structure of the eye-tracking module 412 that tracks the gaze direction of one eye is described separately in the embodiments of this application.

[0160] For example, such as Figure 10 The diagram shown is an exemplary schematic of the filter on the sensor of the light source 4121 and the camera module 4122 in the eye-tracking module 412 of this application embodiment. The eye-tracking module 412 includes a light source 4121 and a camera module 4122, wherein the camera module 4122 includes a lens and an image sensor 1020, and a filter assembly 1021 is disposed on the image sensor 1020. In some embodiments, the filter assembly 1021 can be a physical device that filters light by physical means. Figure 10 In the example shown, the filter component 1021 is formed by etching and other processes on a single filter. In some embodiments, the filter component 1021 may include multiple filters.

[0161] In some embodiments, such as Figure 10As shown in (a), the eye-tracking module 412 includes eight light sources 4121, namely: light source 1 to light source 8. These eight light sources 4121 include four types of light sources, forming four light source groups, with each group containing light sources of the same type. In some implementations, a light source can emit one type of light; typically, a light source will not emit more than two types of light. In some embodiments, different types of light can be light of different wavelengths, allowing N different types of sub-regions to filter light of different wavelengths. In other embodiments, different types of light can also be light with different modulations, for example, different types of light with different vibration directions, allowing N different types of sub-regions to filter light with different vibration directions. In some implementations, different types of sub-regions can be formed on the same optical element using different etching processes. For example, when different types of sub-regions can be formed by different filters, different etching patterns can be formed on different sub-regions on a large filter using etching processes to create different filtering effects in different sub-regions, thereby allowing different sub-regions to transmit light of different wavelengths. For ease of explanation, the following explanation will use the example of different types of light having different wavelengths. For example, light source 1 and light source 5 form a light source group, and the wavelength of light emitted by light source 1 and light source 5 is a; light source 2 and light source 6 form a light source group, and the wavelength of light emitted by light source 2 and light source 6 is b; light source 3 and light source 7 form a light source group, and the wavelength of light emitted by light source 3 and light source 7 is c; light source 4 and light source 8 form a light source group, and the wavelength of light emitted by light source 4 and light source 8 is d.

[0162] like Figure 10 As shown in (b), the filter assembly 1021 is composed of filter units 1022. For example, the filter assembly 1021 is composed of an array of multiple filter units 1022, and each filter unit 1022 is composed of four sub-regions 1023 that can filter light of wavelengths a, b, c, and d, respectively. In this embodiment, filtering light of a certain wavelength can mean that the sub-region can only pass light of that wavelength. For example, filtering a sub-region of light of wavelength a means that the sub-region can only pass light of wavelength a.

[0163] For example, such as Figure 11 As shown, it is based on Figure 10 As shown in (a), eight light sources 4121 illuminate the human eye, consisting of... Figure 10 An exemplary schematic diagram of a human eye spot image acquired by the camera module 4122 with the filter shown in (b). The human eye spot image includes the outline of the human eye 901 and the iris area 902 includes four spots: spot 1, spot 2, spot 3 and spot 4.

[0164] Because the camera module 4122 employs a filter assembly 1021 comprising sub-regions that filter wavelengths a, b, c, and d respectively, the light spot in the human eye light spot image obtained by the camera module 4122 will contain bright and dark sub-regions. The dark sub-regions are formed because no corresponding light (e.g., infrared light of the corresponding wavelength) passes through the sub-region in the filter assembly 1021 and illuminates the image sensor 1020. The bright sub-regions are formed because corresponding light (e.g., infrared light of the corresponding wavelength) passes through the sub-region in the filter assembly 1021 and illuminates the image sensor 1020. In some embodiments, since different types of light are light of different wavelengths, the human eye light spot image can be captured in a single video frame, allowing for fast and accurate light source matching. In some embodiments, the image sensor has an array of pixels, which can be arranged in a specific manner on the image sensor. The image sensor can be a color image sensor or a monochrome image sensor. In some embodiments, the position, size, and / or shape of the sub-regions correspond to the pixels of the image sensor 1020, ensuring that the presence or absence of corresponding light transmission in each sub-region can be reflected on the image sensor 1020, which helps improve the accuracy of eye tracking. Figure 11 For ease of explanation, this example illustrates a sub-region whose position matches the position of a pixel in the image sensor 1020, and whose size and shape are identical to the corresponding pixel. In other embodiments, the position, size, and / or shape of the sub-region may differ from those of a pixel in the image sensor 1020.

[0165] exist Figure 11 In the image, each sub-region of the filter component 1021 is mapped to the iris region 902 containing the light spot in the human eye image. Each sub-region corresponds to one pixel, at least within the iris region. In each corresponding sub-region of the human eye light spot image, pixels with pattern filling (such as a shaded area A1) indicate that the corresponding sub-region transmits light of the corresponding wavelength, while pixels without pattern filling indicate that the corresponding sub-region does not transmit light of the corresponding wavelength.

[0166] For example, light spot 1 is formed in image sensor 1020 by light transmission and filtering component 1021 with wavelength a. The area where light spot 1 is located corresponds to multiple sub-regions, such as... Figure 11As shown, the area containing spot 1 corresponds to multiple sub-regions with filtering wavelengths a, b, c, and d. Sub-regions with filtering wavelengths b, c, and d have no light transmission, while sub-regions with filtering wavelength a have light transmission at wavelength a. Therefore, the pixels corresponding to the multiple sub-regions with filtering wavelengths a (such as the shaded area A1) are bright, thus forming spot 1. Similarly, spot 2 is formed by the light transmission filtering component 1021 with wavelength d, spot 3 is formed by the light transmission filtering components 1021 with wavelengths c and b, and spot 4 is formed by the light transmission filtering component 1021 with wavelength a. Therefore, the order of wavelengths forming spots 1-4 in the human eye spot image is wavelength a, wavelength d, wavelength c and wavelength b, wavelength a.

[0167] Generally, the spatial position of a light spot corresponds to the spatial position of a light source, and the relative positions between light spots are generally consistent with the relative positions between light sources. This allows for easy and accurate identification of which light source each of the light spots 1-4 corresponds to based on the human eye's image of the light spots. Furthermore, the location at which pixels the image sensor 1020 can obtain reflected light from which light sources is related to the eye's gaze direction, thus allowing for accurate determination of the gaze point of the glasses.

[0168] It should be noted that the specific spatial correspondence between the light spot and the light source is related to the optical structure of the lens set according to actual needs. For example, in some embodiments, the light spot near the bottom in the human eye's light spot image is generally obtained by illumination and reflection from a light source closer to the ground; in other embodiments, there are also optical structures that can achieve the effect that the light spot near the bottom in the human eye's light spot image is generally obtained by illumination and reflection from a light source farther from the ground, which is not limited here.

[0169] In Figure 11 In the illustrated embodiment, the light spots near the bottom of the human eye's light spot image are generally obtained by reflection from a light source closer to the ground. Therefore, it can be quickly and accurately determined that: light spot 1 is obtained by light source 1, therefore, light spot 1 matches light source 1; light spot 2 is obtained by light source 8, therefore, light spot 2 matches light source 8; light spot 3 is obtained by light sources 7 and 6, therefore, light spot 3 matches light sources 7 and 6; and light spot 4 is obtained by light source 5, therefore, light spot 4 matches light source 5. That is, light spots 1, 2, 3, and 4 match light sources 1, 8, 7, 6, and 5, respectively.

[0170] Thus, adopting Figure 10The eye-tracking module 412 shown acquires images of light spots in the human eye, which can more quickly and accurately eliminate ambiguity in the matching process between the light source and the light spot, and accurately distinguish which light spot corresponds to which light source(s), thereby helping to determine the user's gaze point more quickly and accurately in the future.

[0171] In the above embodiment, based on the type of light that can be filtered by the corresponding sub-region in the filter unit for the bright sub-region in the light spot, combined with the type of light emitted by the light sources, the relative positions of the light spots in the human eye light spot image and the relative positions of each light source, the light spots in the human eye light spot image and the light sources in the eye tracking module 412 are matched.

[0172] It is understandable that in some embodiments, if the types of light emitted by the light sources are different from each other, the matching of the light source and the light spot can be completed simply by combining the type of light that the bright sub-region in the light spot can filter in the corresponding sub-region in the filter unit with the type of light emitted by each light source.

[0173] In some embodiments, if the light source includes two or more (including two) light sources emitting the same type of light, the light sources emitting the same type of light can be controlled to emit light at different times, or to emit light at different frequencies, to further distinguish the light sources. For example, light sources emitting the same type of light emit light alternately at a first frequency (i.e.,), and a camera module acquires X human eye spot images at a second frequency greater than or equal to the first frequency, where X is greater than or equal to 2, and the X human eye spot images include human eye spot images acquired by the camera module when each of the light sources emitting the same type of light emits light separately. For example, if using... Figure 10 The light sources shown in (a) include light source 1 and light source 5, both emitting light of wavelength 'a'. Light source 1 and light source 5 can be controlled to alternately emit light at a frequency of 1 ms per emission, and human eye spot images can be acquired every 0.5 ms. The resulting X human eye spot images will include human eye spot images when light source 1 emits light alone, and also human eye spot images when light source 5 emits light alone. By combining these X human eye spot images and the light sources that emitted light when acquiring each human eye spot image, and based on the type of light that can be filtered by the corresponding sub-region in the filter unit for the bright sub-region in the spot, and by combining some or all of the information such as the type of light emitted by each light source, the relative positions of spots in the human eye spot image, and the relative positions of each light source, the matching of light sources and spots can be completed more accurately.

[0174] The above description, using a VR scenario as an example, illustrates the process of eye tracking in a VR head-mounted display device 100 that includes an eye-tracking module 412.

[0175] For ease of description, in some embodiments of this application, the light source 4121 and camera module 4122 used to match the light source with the light spot in the eye-tracking module 412 are referred to as eye-tracking devices. This eye-tracking device can be used as a whole component in other electronic devices in other scenarios, such as mobile smart devices, large-screen display devices, smart vehicle devices, etc., and is not limited here.

[0176] Understandable, Figure 10 The number and arrangement of light sources 4121 in the eye-tracking device shown, as well as the arrangement of filter units 1022 and sub-regions 1023 in the filter assembly 1021 on the image sensor 1020 of the camera module 4122, are only examples.

[0177] In some embodiments, the eye-tracking device includes M light sources 4121 and a camera module 4122, where M is a positive integer greater than or equal to 2, and each light source emits one type of light. These M light sources emit N different types of light to the user's eyes, where N is a positive integer greater than or equal to 2. That is, the M light sources can emit two or more types of light, and different light sources can emit the same type of light.

[0178] The camera module 4122 includes a filter assembly 1021 and an image sensor 1020. The filter assembly 1021 is configured to filter at least a portion of the light reflected from the eye after being illuminated by M light sources 4121. The filter assembly 1021 includes multiple filter units 1022, each filter unit 1022 comprising at least N sub-regions 1023. Each filter unit 1022 filters at least N different types of light, and each sub-region 1023 filters one type of light. The image sensor 1020 is configured to acquire the filtered at least a portion of the light to obtain an image of the light spot in the human eye. The filter assembly 1021 may include sub-regions capable of filtering the N types of light emitted by the M light sources, and may also include sub-regions capable of filtering even more types of light.

[0179] Before the light from a light source is captured by the image sensor 1020 after being reflected by the human eye, the filter assembly 1021 filters the reflected light. Since each light source emits a corresponding type of light (e.g., wavelength or modulation), and each sub-region can transmit the corresponding type of light, when light from a certain light source reaches the filter assembly, if the light from that light source matches a certain sub-region, the light from that light source can pass through that sub-region and form a light spot on the image sensor 1020. Therefore, the light spot on the human eye's light spot image can reflect which type or types of light formed it, which helps improve the accuracy and efficiency of light spot matching and enhances the precision of the human eye's gaze point determined by eye tracking.

[0180] In some embodiments, to avoid the light emitted by the light source 4121 affecting the human eye, the M light sources 4121 can be invisible light sources. These invisible light sources may include near-infrared light sources, and / or far-infrared light sources, and / or ultraviolet light sources. For example, the multiple invisible light sources 4121 can be infrared light sources, ultraviolet light sources, or a subset of them may be infrared light sources and a subset of them may be ultraviolet light sources; this is not limited here. When the invisible light source 4121 is an infrared light source, it can be a near-infrared light source, a far-infrared light source, or a subset of them may be near-infrared light sources and a subset of them may be far-infrared light sources; the specific selection can be made according to the actual application requirements, and this is not limited here.

[0181] In some embodiments, a light source emits a type of light, which is light of a single wavelength, and different types of light have different wavelengths. For example, one type of light may have a wavelength of 800 nm, while another type of light may have a wavelength of 900 nm.

[0182] In some embodiments, one type of light is continuous light within a wavelength range, or light within two or more (including two) discontinuous wavelength ranges, and the wavelength ranges of different types of light do not include light of the same wavelength. For example, one type of light may have a wavelength of 800nm ​​to 850nm, and another type of light may have a wavelength of 900nm to 950nm; or, one type of light may have wavelengths of 800nm ​​to 830nm and 850nm to 880nm, and another type of light may have wavelengths of 900nm to 930nm and 950nm to 980nm, without limitation.

[0183] In some embodiments, only a portion of a type of light can pass through a sub-region that filters that type of light. For example, if the wavelength of a type of light is 800nm ​​to 850nm, then the sub-region that filters that type of light can filter only light with a wavelength of 820nm, while blocking light with other wavelengths; or, if the wavelength of a type of light is 800nm ​​to 850nm, then the sub-region that filters that type of light can filter only light with a wavelength of 820nm to 830nm, while blocking light with other wavelengths, and so on.

[0184] In some embodiments, all light of a certain type can pass through a sub-region of filtering that type of light. For example, if the wavelength of a certain type of light is 800nm ​​to 850nm, then the sub-region of filtering that type of light can filter light with wavelengths of 800nm ​​to 850nm, while preventing light of other wavelengths from passing through.

[0185] It is understandable that the specific number, wavelength distribution, and arrangement of the light sources 4121 in the eye-tracking device can be in many different ways. The camera module 4122 can be placed in many different positions relative to the light sources 4121, and there can be multiple camera modules 4122. Similarly, there can be many ways to arrange the sub-regions in the filter unit of the filter assembly.

[0186] The following example, using light of one wavelength as an example, illustrates the various possible ways to achieve this:

[0187] (1) The following is an example illustrating the number, wavelength distribution, and arrangement of the light sources 4121 in the eye-tracking device:

[0188] Each light source emits light of one wavelength. Since the M light sources need to include light sources that can emit light of at least two different wavelengths, the number of the M light sources is at least 2.

[0189] It is understandable that the number of M light sources can also be 3, 4, 5, 6, 7, 8, 9, 10 or more, and is not limited here. With more light sources, the human eye can be illuminated more comprehensively from more different angles, resulting in a larger number of light spots in the human eye's light spot image, thus making the final calculated gaze point more accurate. However, the more light sources there are, the greater the energy consumption. Therefore, electronic devices using this eye-tracking device need to select an appropriate number of light sources based on the actual application requirements.

[0190] To facilitate understanding, the following example uses 8 M light sources to illustrate the wavelength distribution of these M light sources:

[0191] The M light sources include at least two wavelengths:

[0192] In some embodiments, each light source emits a different wavelength than the others. That is, these eight light sources include eight different wavelengths. Since the wavelengths of the light emitted by the light sources are all different, the light source emitting that wavelength can be directly determined based on the wavelength of the light that forms a spot in the human eye's light spot image. This allows for accurate, fast, and unambiguous matching of light source spots, improving the accuracy of eye tracking in determining the human eye's gaze point.

[0193] However, if the light emitted by the light source has too many wavelengths, then the number of sub-regions in the filter unit used to filter these wavelengths separately will also increase, resulting in an excessively large filter unit area. Consequently, most light cannot pass through the filter unit, which is detrimental to obtaining a complete light spot. Therefore, in some embodiments, the plurality of invisible light sources includes at least two light sources emitting the same wavelength. Reducing the number of different types of sub-regions correspondingly reduces the area of ​​the filter unit, allowing for a more accurate determination of the shape and size of the reflected light spot, thereby improving the accuracy of eye tracking in determining the gaze point.

[0194] For example, light sources emitting the same wavelength are called a group of light sources. The more groups of sources there are, the faster the matching relationship between the light spot and the light source can be determined. The fewer groups of light sources there are, the more complete the light spot can be obtained.

[0195] In some embodiments, these eight light sources (light source 1, light source 2, light source 3, light source 4, light source 5, light source 6, light source 7, and light source 8) can emit light of seven different wavelengths (wavelength a, wavelength b, wavelength c, wavelength d, wavelength e, wavelength f, and wavelength g), i.e., there are seven groups of light sources. One of the groups of light sources contains two light sources (for example, light source 8 and light source 1 can both emit light of wavelength a).

[0196] In some embodiments, these eight light sources (light source 1, light source 2, light source 3, light source 4, light source 5, light source 6, light source 7, and light source 8) can emit light of six different wavelengths (wavelength a, wavelength b, wavelength c, wavelength d, wavelength e, and wavelength f), i.e., there are six groups of light sources. One group may contain three light sources (for example, light sources 7 and 8, along with light source 1, can both emit light of wavelength a), or two groups may each contain two light sources (for example, light source 8, along with light source 1, can both emit light of wavelength a, and light source 7, along with light source 2, can both emit light of wavelength b).

[0197] In some embodiments, these eight light sources (light source 1, light source 2, light source 3, light source 4, light source 5, light source 6, light source 7, and light source 8) can emit light of five different wavelengths (wavelength a, wavelength b, wavelength c, wavelength d, and wavelength e), i.e., there are five groups of light sources. One group of light sources may contain four light sources (e.g., light sources 6, 7, and 8 can all emit light of wavelength a along with light source 1), another group may contain three light sources, and another group may contain two light sources (e.g., light sources 7 and 8 can all emit light of wavelength a along with light source 1, and light sources 6 and 2 can both emit light of wavelength b), or all three groups may contain two light sources (e.g., light source 8 can all emit light of wavelength a along with light source 1, light sources 7 and 2 can both emit light of wavelength b, and light sources 6 and 3 can both emit light of wavelength c).

[0198] In some embodiments, these eight light sources can emit light of four different wavelengths, three different wavelengths, two different wavelengths, etc., without limitation.

[0199] These eight light sources can be arranged in various different ways:

[0200] In some embodiments, these eight light sources are on the first surface (e.g. Figure 2A The end of the lens tube closest to the human eye is arranged in a centrally symmetrical, regular shape. This centrally symmetrical, regular shape avoids blind spots in illuminating the human eye, ensuring that light sources reach the eye from all angles, reducing the probability of ambiguous light sources, and thus improving the accuracy of eye tracking in determining the eye's gaze point.

[0201] For example, such as Figure 10 As shown in (a) above, this is an example of eight light sources arranged evenly in a circle. An example is... Figure 12 As shown in (a) above, this is an example of eight light sources arranged uniformly in a rectangle. An example is... Figure 12 As shown in (b) in the figure, there is an example of eight light sources arranged in a diamond shape.

[0202] It is understood that in other embodiments, the eight light sources are arranged evenly in other centrally symmetrical regular patterns, which is not limited here.

[0203] In some embodiments, the eight light sources are arranged in an irregular shape on the first surface. This irregular arrangement allows for various variations in the light path from the light source illuminating the human eye to the camera module, resulting in light spots at different positions in the human eye. This enables more accurate determination of the human eye's movement state, thereby improving the accuracy of eye tracking in determining the eye's gaze point.

[0204] For example, such as Figure 13 As shown in (a) above, this is an example of eight invisible light sources arranged in an irregular shape. Examples include... Figure 13 As shown in (b), this is an example of eight invisible light sources arranged in another irregular shape.

[0205] In some embodiments, the center of the M light source arrangement is on the same horizontal line as the center of the human eyeball when using the eye-tracking device.

[0206] It is understandable that the spacing between the M light sources can be set according to their arrangement and the distance between the invisible light source and the eyeball when using the eye-tracking device; no limitation is made here.

[0207] In some embodiments, when multiple light sources emitting the same wavelength exist, these light sources are not adjacent. In some embodiments, the distance between these multiple light sources emitting the same wavelength is greater than the minimum distance between light sources emitting different wavelengths. For example, ... Figure 10 As shown in (a), the distance between light source 1 and light source 5, which emit light of the same wavelength a, is greater than the distance between light source 8, which emits light of wavelength d, and light source 7, which emits light of wavelength c. This reduces the probability of ambiguity when matching light source spots and improves the accuracy of eye tracking in determining the gaze point of the human eye.

[0208] The following example uses eight light sources arranged in a uniform circular pattern on a surface. Combining this with several wavelength distribution methods, several examples of light source arrangement structures in eye-tracking devices are provided:

[0209] For example, such as Figure 14 As shown in (a), eight invisible light sources can be divided into four groups in pairs: light sources 1 and 5, light sources 2 and 6, light sources 3 and 7, and light sources 4 and 8. Two light sources in each group emit light of the same wavelength, and any two groups of the four light sources emit light of different wavelengths.

[0210] For example, such as Figure 14 As shown in (b), the eight invisible light sources can be divided into three groups: [light source 1, 4 and 6], [light source 2, 5 and 7], and [light source 3 and 8]. Each group of light sources emits light of the same wavelength, and any two groups of light sources emit light of different wavelengths.

[0211] For example, such as Figure 14 As shown in (c), the eight invisible light sources can be divided into two groups according to [light source 1, 3, 5 and 7] and [light source 2, 4, 6 and 8], with each group emitting light of the same wavelength and the two groups emitting light of different wavelengths.

[0212] (2) The following example illustrates the number and arrangement of camera modules 4122 in the eye-tracking device:

[0213] The number of camera modules 4122 is at least one.

[0214] In some embodiments, the number of camera modules 4122 can be 2, 3, 4, 5 or more, and is not limited here. The more camera modules 4122 there are, the more accurate the light spot matching will be, but it will also bring greater power consumption. Therefore, it is necessary to select an appropriate number of camera modules according to the actual application requirements.

[0215] The camera module 4122 can be located at a position that can capture the light reflected from the eyeball after the light emitted by the M light sources 4121 illuminates it. Specifically, the position of the camera module 4122 can be determined based on the number and arrangement of the light sources, the position of the human eyeball when using the eye-tracking device, etc., and is not limited here.

[0216] In some embodiments, the camera module 4122 may be located on the same plane as the M light sources 4121. For example, in Figure 2A The end of the end tube closest to the human eye.

[0217] In some embodiments, the camera module 4122 may be located outside the geometry formed by the M light sources. This prevents light from the display screen from being blocked from entering the human eye through the lens.

[0218] For example, consider an eye-tracking device with eight light sources evenly distributed in a circular pattern and only one camera module. Figure 15 As shown, this camera module can be set at any of the positions shown in the diagram: position 1, position 2, position 3, position 4, position 5, and position 6.

[0219] In some embodiments, in order to obtain more comprehensive light reflected from the light source that illuminates the human eye, the camera module 4122 may also be located within the geometric shape formed by the M light sources, or may not be on the same plane as the M light sources 4121, thereby improving the accuracy of eye tracking in determining the gaze point of the human eye. This is not limited here.

[0220] (3) The following is an example illustrating the arrangement of sub-regions in the filter unit 1022 of the camera module 4122:

[0221] It is understandable that if the light source in the eye-tracking device can emit N different wavelengths of light, then a filter unit includes N sub-regions that can filter different wavelengths of light, and may also include sub-regions that can filter other wavelengths of light.

[0222] In a filter unit, each sub-region may include only one or multiple regions; this is not limited here.

[0223] There are many ways to arrange the sub-regions in the filter unit. For example, the sub-regions that filter different wavelengths of light can be arranged in order of wavelength or randomly. There is no limitation here.

[0224] The sub-regions within the filter unit can have various shapes, specifically determined by the shape of pixels in the image sensor. Different shaped sub-regions correspond to the shapes of different pixels on the sensor, allowing the sensor to acquire more comprehensive and accurate light filtered by the filter components, improving the accuracy of light spot matching and thus enhancing the accuracy of eye tracking in determining the gaze point. Figure 10 The sub-region 1023 in the filter unit 1022 is shown as a square.

[0225] The number of sub-regions filtering different wavelengths of light in the filter unit can be the same or different; this is not limited here. The size of the sub-regions filtering different wavelengths of light in the filter unit can be the same or different; this is not limited here. The shape of the sub-regions filtering different wavelengths of light in the filter unit can be the same or different; this is not limited here.

[0226] The number, size, and shape of the specific sub-regions filtering each wavelength of light in the filter unit can be set according to the actual application requirements, and are not limited here.

[0227] In some embodiments, the area of ​​each sub-region is greater than or equal to one image sensor pixel and / or less than or equal to five image sensor pixels, so that the presence or absence of corresponding light transmission in each sub-region can be easily acquired and represented by the image sensor. In some embodiments, the area of ​​each sub-region is less than 1 / N of the area of ​​a light spot in a human eye light spot image, where N can be the number of types of light emitted by the light source. For example, Figure 10 The light source shown in (a) can emit light of four wavelengths. Therefore, the area of ​​each sub-region can be less than 1 / 4 of the area of ​​a single spot in the human eye spot image. If the area of ​​a single spot in the human eye spot image is less than 16 pixels, then the area of ​​each sub-region can be less than 2 pixels. In this way, each spot is covered by at least four sub-regions, ensuring that more wavelength information is obtained for each spot, thereby improving the accuracy of eye tracking in determining the gaze point.

[0228] Taking a sub-region as a square as an example, such as Figure 10 As shown in (b), four square sub-regions 1023 are arranged into a larger square, serving as filter units 1022. Multiple filter units 1022 are comprised of a larger filter assembly 1021 that covers the image sensor 1020.

[0229] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0230] While the description of this application is presented in conjunction with some embodiments, this does not mean that the features of this application are limited to this embodiment. On the contrary, the purpose of describing the application in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of this application. To provide a thorough understanding of this application, many specific details will be included in the following description. This application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0231] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0232] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, "connection" can be a detachable connection or a non-detachable connection; it can be a direct connection or an indirect connection through an intermediate medium. The directional terms mentioned in the embodiments of this application, such as "upper," "lower," "left," "right," "inner," and "outer," are only for reference to the directions in the accompanying drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of this application, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application. "Multiple" refers to at least two.

[0233] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. In the above embodiments, implementation can be achieved wholly or partially by software, hardware, firmware, or any combination thereof. When implemented using software, implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0234] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An eye-tracking device, characterized in that, include: M light sources are configured to emit N different types of light to the user's eyes, where M is a positive integer greater than or equal to 2, N is a positive integer greater than or equal to 2, and N is less than or equal to M; The camera module includes a filter assembly and an image sensor; wherein, The light filtering assembly is configured to filter at least a portion of the light reflected after the M light sources illuminate the eye. The light filtering assembly includes a plurality of light filtering units, and each light filtering unit includes at least N different types of sub-regions, wherein the different types of sub-regions filter different types of light. The image sensor is configured to acquire at least a portion of the light transmitted through the filter assembly to obtain a human eye spot image, wherein the at least a portion of the light transmitted through the filter assembly forms at least one spot on the human eye spot image.

2. The eye-tracking device according to claim 1, characterized in that, The image sensor includes an array of pixels, the sub-regions correspond to the pixels, a light spot includes multiple pixels, and a light spot corresponds to the N different types of sub-regions.

3. The eye-tracking device according to claim 2, characterized in that, The area of ​​the sub-region is less than or equal to 5 pixels.

4. The eye-tracking device according to claim 2 or 3, characterized in that, The area of ​​the sub-region is less than or equal to 1 / N of the area of ​​one of the light spots.

5. The eye-tracking device according to claim 2 or 3, characterized in that, Each sub-region corresponds one-to-one with a pixel of the image sensor.

6. The eye-tracking device according to any one of claims 1 to 3, characterized in that, The number of light spots is less than M.

7. The eye-tracking device according to any one of claims 1 to 3, characterized in that, The eye-tracking device also includes a processor for determining the position of the eyes, the processor communicating with the camera module, and the processor being configured to receive eye spot images from the camera module.

8. The eye-tracking device according to any one of claims 1 to 3, characterized in that, M is greater than or equal to 4, and / or N is greater than or equal to 4.

9. The eye-tracking device according to any one of claims 1 to 3, characterized in that, N is less than M, and at least two of the light sources emit the same type of light, wherein the two light sources emitting the same type of light are not adjacent.

10. The eye-tracking device according to claim 9, characterized in that, The M light sources include multiple light source groups, and each light source group includes at least two light sources that emit the same type of light, while the light sources in different light source groups emit different types of light.

11. The eye-tracking device according to claim 9, characterized in that, Two light sources that emit the same type of light have different emission frequencies or emission times.

12. The eye-tracking device according to any one of claims 1 to 3, 10 or 11, characterized in that, The M light sources are invisible light sources.

13. The eye-tracking device according to claim 12, characterized in that, The M invisible light sources include at least one or more of the following: near-infrared light sources, far-infrared light sources, or ultraviolet light sources.

14. The eye-tracking device according to any one of claims 1 to 3, 10, 11 or 13, characterized in that, The aforementioned type of light is light of one wavelength, light that is continuous within one wavelength range, or light of two or more discontinuous wavelength ranges.

15. The eye-tracking device according to any one of claims 1 to 3, 10, 11 or 13, characterized in that, The sub-region filtering a type of light specifically includes: light of some or all wavelengths of a type of light passing through the sub-region.

16. The eye-tracking device according to any one of claims 1 to 3, 10, 11 or 13, characterized in that, The eye-tracking device is a virtual reality wearable device, which includes an end face facing the eyes, and the M light sources are located on the end face.

17. The eye-tracking device according to claim 16, characterized in that, The virtual reality wearable device also includes a lens assembly, which includes at least one optical device. One side of the lens assembly faces the eye, and the M light sources are arranged around one side of the lens assembly.

18. The eye-tracking device according to claim 17, characterized in that, The M light sources are arranged at equal intervals around one side of the mirror assembly.

19. The eye-tracking device according to claim 17 or 18, characterized in that, The camera module is located on the end face and is arranged around one side of the lens group; or, The camera module is located on the side of at least one optical device that is away from the end face.

20. The eye-tracking device according to any one of claims 1 to 3, 10, 11, 13, 17 or 18, characterized in that, The center of the camera module does not coincide with the center of the M light sources.

21. An eye-tracking method, characterized in that, The method includes: M light sources emit N different types of light toward the user's eyes, where M is a positive integer greater than or equal to 2, N is a positive integer greater than or equal to 2, and N is less than or equal to M; The camera module's filter assembly filters at least a portion of the light reflected after the M light sources illuminate the eye; the filter assembly includes multiple filter units, each filter unit comprising at least N different types of sub-regions, wherein different types of sub-regions filter different types of light; The image sensor in the camera module acquires at least a portion of the light that passes through the filter assembly to obtain a human eye spot image, wherein the at least a portion of the light that passes through the filter assembly forms at least one spot on the human eye spot image.

22. The method according to claim 21, characterized in that, The sub-regions correspond to the pixels of the image sensor, and one light spot includes multiple pixels. One light spot corresponds to the N different types of sub-regions.

23. The method according to claim 22, characterized in that, The area of ​​the sub-region is less than or equal to 5 pixels of the image sensor.

24. The method according to claim 22 or 23, characterized in that, The area of ​​the sub-region is less than or equal to 1 / N of the area of ​​one of the light spots.

25. The method according to claim 22 or 23, characterized in that, Each sub-region corresponds one-to-one with a pixel of the image sensor.

26. The method according to any one of claims 21 to 23, characterized in that, The eye's gaze point is obtained by acquiring the light spot image of the human eye from the image sensor.

27. The method according to any one of claims 21 to 23, characterized in that, The number of light spots is less than M.

28. The method according to any one of claims 21 to 23, characterized in that, The filtered light at least partially forms at least one spot on the human eye's light spot image, the area of ​​the sub-region being less than or equal to 1 / N of the area of ​​the spot.

29. The method according to any one of claims 21 to 23, characterized in that, N is less than M, and at least two of the light sources emit the same type of light, wherein the two light sources emitting the same type of light are not adjacent.

30. The method according to claim 29, characterized in that, The M light sources include multiple light source groups, and each light source group includes at least two light sources that emit the same type of light, while the light sources in different light source groups emit different types of light.

Images