Eyewear with eye movement tracking system

Abstract

An eyewear (100) includes: a frame (110) comprising a front frame (111), two temples (112, 113) connected to the front frame (111), and at least one lens (114, 115) connected to the front frame (111); an eye-tracking system (200) associated with the frame (110); characterized in that the eye-tracking system (200) includes: at least one infrared radiation source (210, 211) connected to the frame (110) to face and emit infrared radiation toward the corresponding eye to be tracked when the user wears the eyewear (100); and a plurality of infrared photodetectors (220) connected to the at least one lens (114, 115). At the edge of 115), the infrared photodetector (220) is configured to detect optical signals reflected by the eye to be tracked and generate corresponding electrical detection signals; a processing and control unit (230) is associated with the at least one infrared radiation source (210, 211) and the plurality of infrared photodetectors (220), the processing and control unit (230) being configured to enable the at least one infrared radiation source (210, 211) to emit infrared radiation, receive electrical detection signals and estimate the gaze direction of the eye to be tracked based on the received electrical detection signals; a battery (240) is associated with the processing and control unit (230), the infrared photodetectors (220) and the at least one infrared radiation source (210, 211) to power them.

Description

Technical Field

[0001] This invention relates to eye-wearing devices, and more particularly to smart eye-wearing devices including an eye-tracking system. Background Technology

[0002] Smart eyewear encompasses any type of device designed to be worn on at least one of the wearer's eyes and having a frame configured to be worn on the head, including any headband device.

[0003] As is known, smart eye wearables include electronic devices adapted to function as electronic calculators and capable of supporting activities performed by users wearing these devices.

[0004] More specifically, electronic devices are capable of playing audio and video, launching and running software applications, and recognizing shapes and objects placed in their surrounding environment.

[0005] More specifically, the electronic devices typically include processing and control units, as well as sensors, integrated into the eyeglass frames or temples.

[0006] For example, it is known to provide eye-tracking systems to smart eyewear in order to enable / disable some functions of the smart eyewear, such as for augmented reality and mixed reality, and / or to monitor eye movement for medical, psychological, commercial and marketing purposes, such as for tracking the direction of gaze toward a store window or supermarket shelf.

[0007] Several different types of eye-tracking systems are currently known.

[0008] The most widely used first-type eye-tracking system consists of one or more sources for illuminating the eye being tracked and one or more cameras for capturing the movement of the eye. However, such systems are bulky and therefore cannot be embedded in smart eyewear that must look the same as ordinary eyewear.

[0009] Given this drawback, a second type of eye-tracking system becomes more attractive, which includes an infrared and / or near-infrared wavelength source illuminating the eye to be tracked and a detector for detecting the optical signal reflected by the illuminated eye; in these systems, the gaze direction is obtained based on the detected optical signal. This is because these systems operate at infrared and / or near-infrared wavelengths outside the sensitivity range of the human eye and therefore do not interfere with visual abilities.

[0010] In this specification, infrared wavelength is intended to indicate a wavelength between 700 nm and 1 mm.

[0011] Near-infrared wavelengths are intended to indicate wavelengths between 700 nm and 10 µm.

[0012] US 10,521,658 B2, US 11,256,086 B2, US10,606,071 B1, and US 10,481,687 B1 describe examples of a second type of eye-tracking system that uses optical elements (such as dichroic mirrors or gratings) to couple light reflected from the eye to a detector. This involves very complex and very expensive manufacturing of such systems. Furthermore, such eye-tracking systems provide substantial modifications to the appearance of eye-wearing devices. Summary of the Invention

[0013] The purpose of this invention is to overcome the above-mentioned disadvantages, and in particular to design an eye-wearing device with an eye-tracking system that is simpler to manufacture and cheaper than the prior art.

[0014] Another object of the present invention is to design an eye-wearing device with an eye-tracking system that has the same appearance as a normal non-smart eye-wearing device.

[0015] This and other objectives according to the invention are achieved by manufacturing an eye-wearing device with an eye-tracking system as described in claim 1.

[0016] Other features of eye-wearing devices with eye-tracking systems are the subject of the dependent claims. Attached Figure Description

[0017] The characteristics and advantages of the eye-wearing device with an eye-tracking system according to the invention will become more apparent from the following exemplary but non-limiting description with reference to the accompanying schematic diagrams, in which: - Figure 1 This is a schematic perspective view of an eye-wearing device according to the present invention; - Figure 2 This is a schematic perspective view of a lens included in an eyewear device according to the present invention and connected to a plurality of photodetectors; - Figure 3 This is a schematic perspective view illustrating the operation of the eyewear according to the present invention; - Figure 4 This is a block diagram illustrating the electronic device included in an eye-wearing device according to the present invention; - Figure 5a is a schematic perspective view of a lens connected to multiple photodetectors for experimental testing; - Figure 5b is a schematic representation of the experimental setup used to test the lens shown in Figure 5a; - Figure 5c illustrates the effect of changing the orientation of the movable IR LED included in the experimental setup of Figure 5b (i.e., by changing the azimuth angle (θ) and yaw angle relative to the vertical axis aligned with the center of the lens in the polar coordinate system shown in Figure 5b). (As shown in Figure 5b) Experimental diagram of light collected by a photodetector. Detailed Implementation

[0018] Referring to the accompanying drawings, an eye-wearing device is shown, which is generally referred to as 100.

[0019] The eyewear 100 includes a frame 110, which includes a front frame 111, two temples 112 and 113 connected to the front frame 111, and at least one lens 114 and 115 connected to the front frame 111. For example, the eyewear is eyeglasses, and in this case, the eyewear may include two lenses; the eyewear 100 may also be a ski mask or a face mask-style sunglasses, and in this case, the eyewear may include a single lens.

[0020] The eyewear 100 advantageously includes an eye-tracking system 200 associated with a frame 110. The eye-tracking system 200 includes at least one infrared radiation source 210, 211, which is coupled to the frame 110 to face the user’s corresponding eye to be tracked and to emit infrared radiation toward that eye when the user wears the eyewear 100.

[0021] In the illustrated embodiment, there are two infrared radiation sources 210 and 211, one for each corresponding eye.

[0022] However, the eye to be tracked may be only one of the user's two eyes, and in this case, the eye-tracking system 200 preferably includes only one infrared radiation source.

[0023] In any case, the at least one infrared radiation source 210, 211 must be pointed at the eye to be tracked.

[0024] For example, the at least one infrared radiation source 210, 211 is an LED.

[0025] The eye-tracking system 200 also includes a plurality of infrared photodetectors 220 optically coupled to the at least one lens 114, 115.

[0026] Optical coupling means that the infrared photodetector is arranged to allow light to be transmitted from the at least one lens 114, 115 to the infrared photodetector 220.

[0027] Specifically, the infrared photodetector may be mechanically coupled to the edge of the at least one lens 114, 115 or coupled to the inner edge of the front frame 111 of the frame 110 facing the edge of the at least one lens 114, 115.

[0028] The infrared photodetector 220 is configured to detect an optical signal reflected by the eye to be tracked, the optical signal being an infrared wavelength, preferably a near-infrared wavelength, and more preferably a wavelength of about 850 nm; the infrared photodetector 220 is also configured to generate a corresponding electrical detection signal.

[0029] For example, the infrared photodetector 220 can be manufactured as a solid-state semiconductor photodetector, such as a photodiode. For example, the infrared photodetector 220 can be made of silicon or other semiconductors, or other materials (such as solution-processable materials).

[0030] The electrical detection signal generated by the infrared photodetector 220 can be an analog signal.

[0031] Specifically, the electrical detection signal generated by the infrared photodetector 220 can be an analog current detection signal.

[0032] The infrared photodetector 220 can be connected to an analog signal converter circuit 222, which converts the analog current detection signal into an analog voltage detection signal. The output of the analog signal converter circuit 222 can be connected to an analog-to-digital converter (ADC) 221, which is configured to convert the analog electrical detection signal into a digital electrical detection signal.

[0033] More generally, the presence of analog signal converter circuit 222 and ADC converter 221 is independent of each other. In fact, in some embodiments, analog signal converter circuit 222 is absent, and infrared photodetector 220 can be directly connected to ADC converter 221. In other embodiments, analog signal converter circuit 222 is present, but ADC converter 221 is absent. In still other embodiments, neither analog signal converter circuit 222 nor ADC converter 221 is present.

[0034] Furthermore, the analog signal converter circuit 222 (if present) and the ADC converter (if present) may be embedded in the photodetector 220, or they may be external relative to the photodetector 220. If the analog signal converter circuit 222 and the ADC converter 221 (if present) are embedded in the photodetector 220, the electrical detection signal generated by the infrared photodetector 220 is a digital signal.

[0035] In the eyewear according to the invention, the at least one lens 114, 115 acts as an optical waveguide that guides a portion of the infrared light incident on the lens after being reflected by the eye irradiated by the at least one infrared radiation source 210, 211 toward the edge. Therefore, the infrared photodetector 220 is configured to detect the optical signal reflected by the (tracked) eye toward the lens and guided within the lens.

[0036] The eye-tracking system 200 also includes a processing and control unit 230 associated with the at least one infrared radiation source 210, 211 and the plurality of infrared photodetectors 220. In particular, the processing and control unit 230 is configured to enable the at least one infrared radiation source 210, 211 to emit infrared radiation, receive electrical detection signals, and estimate the gaze direction of the eye to be tracked based on the received electrical detection signals.

[0037] For example, the processing and control unit 230 is a microcontroller, but more generally, it can be made of analog or digital circuitry.

[0038] If the processing and control unit 230 is made of analog circuitry, then the ADC converter 221 is absent, and the processing and control unit 230 receives analog electrical detection signals. However, if the processing and control unit 230 is made of digital circuitry, then the ADC converter 221 must be present, embedded in the infrared photodetector 220, or external relative to the infrared photodetector, and the processing and control unit 230 receives digital electrical detection signals. Therefore, the electrical detection signals received by the processing and control unit 230 can be either analog or digital signals as previously described. Thus, the term "processing and control unit 230" is used in association with the plurality of infrared photodetectors 220 to indicate that the processing and control unit 230 can be directly or indirectly connected to the infrared photodetectors 220, depending on the possible presence of the analog signal converter circuit 222 and the ADC converter 221.

[0039] Specifically, the processing and control unit 230 may include an internal memory (not shown) in which a computer program is loaded; such a computer program implements an algorithm to estimate the gaze direction of the eye to be tracked based on the received electrical detection signal.

[0040] For example, such an algorithm could be a supervised machine learning algorithm, which can be represented by an artificial neural network (ANN) comprising multiple nodes (called artificial neurons) interconnected by means of branches (called artificial synapses). ANNs will not be described as they are known in the prior art.

[0041] The algorithm implemented in the processing and control unit 230 is preferably trained using a suitably sized dataset consisting of light signal maps. The dataset can consist of simulated data generated with known gaze directions and experimental data. The latter can be automatically acquired, for example, using an artificial eye that has a response in the infrared spectrum similar to that of the human eye. The artificial eye can be mounted on a motor-actuated platform, the position of which can be scanned by a dense grid of points identified by a pair of angles (specifically elevation and azimuth) defining the gaze direction. Training allows the weights of the ANN stored in internal memory to be determined so that eye-tracking estimation can be run in real time with a typical temporal resolution down to the millisecond range.

[0042] Therefore, the training process of the algorithm implemented by the computer program stored in the internal memory of the processing and control unit 230 includes the following steps: - Perform the experimental dataset acquisition step, which includes the following steps: - Provides an eye-wearing device 100 equipped with an eye-tracking system 200; - Provide an artificial eye mounted on a support so that it can rotate about two rotation axes according to multiple attitudes defined by corresponding pitch or tilt angles and corresponding azimuth or yaw angles; - Position the eyepiece 100 in front of the artificial eye; - Scan a dense grid of points in angular space (i.e., composed of pairs of elevation and azimuth angles within the angular range to be estimated by the eye tracker system 200) and acquire electrical detection signals from the infrared photodetector 220 for each point; - The algorithm is trained by using a portion of the acquired experimental dataset (i.e., finding the optimal values ​​of its hyperparameters) and by validating it using another portion of the acquired experimental dataset.

[0043] Once the training process is complete, the optimal, validated parameters (such as the weights of the ANN) are loaded into the computer program.

[0044] The eye-tracking system 200 also includes a battery 240 associated with a processing and control unit 230, an infrared photodetector 220, and the at least one infrared radiation source 210, 211 to power them.

[0045] The eye-tracking system 200 may also include a wireless communication module 250 associated with the processing and control unit 230, which is configured to communicate wirelessly with an electronic terminal (such as a smartphone).

[0046] The estimation of gaze direction and other eye parameters (such as blink or pupil diameter) depends on the number and location of the infrared photodetectors. Optimal locations are obtained through numerical simulation. In particular, the number and location of the at least one infrared radiation source 210, 211 and the infrared photodetector 220 can be varied by means of optical simulation in order to maximize the amount of light collected by the infrared photodetector 220 and its spatial resolution (i.e., the sensitivity of the collected light map to the gaze direction).

[0047] The number of photodetectors 220 determines not only the calculated angular resolution of the gaze direction, but also the values ​​of power consumption and dissipation. In particular, the maximum number of photodetectors 220 results in the maximum angular resolution, but also the maximum values ​​of power consumption and dissipation.

[0048] It is necessary to determine the number of photodetectors, which represents a trade-off between the need for sufficiently high angular resolution and the need for sufficiently low power consumption and dissipation.

[0049] Preferably, there are at least three infrared photodetectors 220.

[0050] More preferably, there are four infrared photodetectors 220.

[0051] Figure 5b shows an example of an experimental setup for validating the results obtained by the described eye-tracking system. This setup provides a standard lens with four infrared semiconductor photodetectors (S1, S2, S3, and S4) attached to the edge of the lens, as shown in Figure 5a. IR light reflected from the eye is simulated by a movable beam generated by an IR LED, which has a narrow emission profile directed towards the lens.

[0052] Figure 5c illustrates the relationship between the azimuth angle (θ) and yaw angle (θ) relative to the vertical axis aligned with the center of the lens in the polar coordinate system shown in Figure 5b. The light signals are collected by four photodetectors as both scan the surface of the lens. These figures illustrate the position sensitivity of the eye-tracking system of the present invention and show examples of data fed as input to an algorithm for estimating the gaze direction.

[0053] For example, these figures show photodetector S1 with θ between -10° and -20° and The photodetector S2 is sensitive to gaze direction between -30° and +30°, while the photodetector S2 is sensitive to gaze direction between +10° and +20°. The gaze direction is sensitive between +30° and +150° and between -45° and -120°, and so on. For each angular region, at least one sensor S1, S2, S3, S4 detects the signal.

[0054] Preferably, the processing and control unit 230 and the battery 240 are installed in two different temples 112, 113. However, they may also be installed in the same temple 112, 113 or in the front frame 111.

[0055] In any case, the interconnection between the infrared photodetector 220, the at least one infrared source 210, the processing and control unit 230, and the battery can be achieved by means of wires or a flexible printed circuit board (PCB).

[0056] In a particular embodiment, battery 240 includes a photovoltaic cell system adapted to recharge battery 240.

[0057] Preferably, the photodetectors have a lateral size between 1 mm and 3 mm, allowing them to be applied to the edges of the at least one lens 214, 215 without protruding beyond its thickness. In this way, no visible changes are introduced to the shape of the frame 210 and the at least one lens 214, 215. The aesthetics of the eyewear 100 are not altered by the introduction of the eye-tracking system 200.

[0058] The Eye Tracking System 200 is compatible with any type of lens in terms of materials, color, and ophthalmic correction.

[0059] Preferably, the infrared photodetector 220 is applied to the edge of the at least one lens 214, 215 by means of an optical adhesive that allows for high optical coupling between the edge of the surface of the at least one lens 214, 215 and the infrared photodetector 220, which is crucial for the operation of the eye-tracking system 200.

[0060] Preferably, the at least one lens 214, 215 has no surface treatments or components designed to increase the light guiding effect of light in the infrared and / or near-infrared wavelengths and / or absorption at the edges. Therefore, the at least one lens 214, 215 is not provided with any additional components (such as mirrors or gratings), such as those described in the prior art, which are designed to increase the light guided toward the edges of the at least one lens in the infrared and / or near-infrared wavelengths, thereby increasing the light received by the infrared photodetector 220. In this way, the production of eye-tracking systems becomes cheaper and simpler compared to the prior art.

[0061] As a first alternative, one or more additional coatings (e.g., interference layers, dielectric dots, or microlenses (e.g., microlenses for myopia control)) can be deposited on the surface of the at least one lens 214, 215 to increase the light guiding effect at infrared and / or near-infrared wavelengths.

[0062] As a second alternative, one or more layers of metamaterial (e.g., metamaterials that are transparent in the visible spectrum and reflective at infrared and near-infrared wavelengths) can be deposited on the surface of the at least one lens 214, 215 to increase the light guiding effect at infrared and / or near-infrared wavelengths.

[0063] As a third alternative, the at least one lens 214, 215 may include a plurality of nanoparticles that absorb incident infrared and / or near-infrared light and re-emit light isotropically with slightly lower energy, thereby increasing absorption at the edges of the at least one lens 214, 215.

[0064] A fourth alternative is to deposit specific masks (also known as coded apertures) on the at least one lens 214, 215. These masks are transparent in the visible spectrum and include patterns that are transparent and opaque to the infrared region in order to improve sensitivity to the orientation of incident light (similar to the operation performed with a scintillator in gamma-ray detection).

[0065] The eye-tracking system 200 can also be incorporated into a head-mounted device. In this case, the head-mounted device includes a frame configured to be worn on the head, at least one lens attached to the frame for positioning in front of at least one eye, and an eye-tracking system 200 associated with the frame, wherein the eye-tracking system and the at least one lens can be manufactured as described above.

[0066] Based on the description provided, the characteristics and relative advantages of the eye-wearing device (the object of this invention) are clear.

[0067] In fact, this type of eyewear incorporates an eye-tracking system that does not require modification of the frame shape of common eyewear because the photodetector is optically coupled to the edge of the lens without protruding beyond its thickness. Furthermore, the eye-tracking system does not require special treatment on the lens surface and can work with ordinary lenses.

[0068] Finally, it is clear that the envisioned eye-wearing device is readily subject to numerous modifications and variations, all of which are within the scope of this invention; furthermore, all details can be replaced by technically equivalent elements. In practice, the materials used and their dimensions can be of any type, depending on the technical requirements.

Claims

1. An eye-wearing device (100), comprising: - A frame (110) comprising a front frame (111), two temples (112, 113) connected to the front frame (111), and at least one lens (114, 115) connected to the front frame (111). - An eye-tracking system (200) associated with the frame (110); The eye-tracking system (200) is characterized in that it comprises: - At least one infrared radiation source (210, 211) is coupled to the eyeglass frame (110) so as to face the corresponding eye to be tracked and emit infrared radiation toward the eye when the user wears the eyewear (100); - A plurality of infrared photodetectors (220), the plurality of infrared photodetectors being optically coupled to the at least one lens (114, 115), the infrared photodetectors (220) being configured to detect optical signals reflected by the eye to be tracked and generate corresponding electrical detection signals; - A processing and control unit (230) associated with the at least one infrared radiation source (210, 211) and the plurality of infrared photodetectors (220), the processing and control unit (230) being configured to enable the at least one infrared radiation source (210, 211) to emit the infrared radiation, receive the electrical detection signal and estimate the gaze direction of the eye to be tracked based on the received electrical detection signal.

2. The eye-wearing device (100) according to claim 1, wherein, The infrared photodetector (220) is mechanically coupled to the edge of the at least one lens (114, 115) or mechanically coupled to the inner edge of the front frame (111) of the frame (110) facing the edge of the at least one lens (114, 115).

3. The eye-wearing device (100) according to claim 1 or 2, comprising a battery (240) associated with the processing and control unit (230), the infrared photodetector (220) and the at least one infrared radiation source (210, 211) for powering them.

4. The eye-wearing device (100) according to one or more of the preceding claims, wherein, The infrared photodetectors (220) optically coupled to the edge of the at least one lens (114, 115) are at least three.

5. The eye-wearing device (100) according to one or more of claims 1 to 4, wherein, There are four infrared photodetectors (220) optically coupled to the edge of the at least one lens (114, 115).

6. The eye-wearing device (100) according to one or more of claims 2 to 5, wherein, The processing and control unit (230) and the battery (240) are installed in two different temples (112, 113).

7. An eye-wearing device (100) according to one or more of the preceding claims, wherein, The processing and control unit (230) includes an internal memory in which a computer program is loaded, the computer program implementing an algorithm to estimate the gaze direction of the eye to be tracked based on the received electrical detection signal, the algorithm being a supervised machine learning algorithm.

8. An eye-wearing device (100) according to one or more of the preceding claims, wherein, The eye-tracking system (200) includes a photovoltaic cell system adapted to recharge the battery (240).

9. An eye-wearing device (100) according to one or more of the preceding claims, wherein, The infrared photodetector (220) is applied to the edge of the at least one lens (214, 215) by means of optical adhesive.

10. An eye-wearing device (100) according to one or more of the preceding claims, wherein, The at least one lens (214, 215) has no surface treatment or component designed to increase the light guiding effect of light at infrared and / or near-infrared wavelengths and / or absorption at the edge.

11. The eye-wearing device (100) according to one or more of claims 1 to 9, wherein, One or more additional coatings, dielectric dots, or microlenses are provided by depositing on the surface of the at least one lens (214, 215) to enhance the light guiding effect at infrared and / or near-infrared wavelengths.

12. The eye-wearing device (100) according to one or more of claims 1 to 9, wherein, One or more layers of metamaterial are provided by deposition on the surface of the at least one lens (214, 215) to enhance the light guiding effect at infrared and / or near-infrared wavelengths.

13. The eye-wearing device (100) according to one or more of claims 1 to 9, wherein, The at least one lens (214, 215) comprises a plurality of nanoparticles that absorb incident infrared and / or near-infrared light and re-emit light isotropically with slightly lower energy, thereby increasing the absorption at the edge of the at least one lens (214, 215).

14. The eye-wearing device (100) according to one or more of claims 1 to 9, wherein, The at least one lens (214, 215) includes an coded aperture mask in the IR spectrum to enhance spatial resolution.

15. A head-mounted device, comprising: - A frame configured to be worn on the head and at least one lens, the at least one lens being attached to the frame to be positioned in front of at least one of the user's eyes; - An eye-tracking system (200) associated with the eye frame; The eye-tracking system (200) is characterized in that it comprises: - At least one infrared radiation source (210, 211) is coupled to the eyeglass frame (110) so as to face the corresponding eye to be tracked and emit infrared radiation toward the eye when the user wears the eyewear (100); - A plurality of infrared photodetectors (220) optically coupled to the edge of the at least one lens (114, 115), the infrared photodetectors (220) being configured to detect optical signals reflected by the eye to be tracked and generate corresponding electrical detection signals; - A processing and control unit (230) associated with the at least one infrared radiation source (210, 211) and the plurality of infrared photodetectors (220), the processing and control unit (230) being configured to enable the at least one infrared radiation source (210, 211) to emit the infrared radiation, receive the electrical detection signal and estimate the gaze direction of the eye to be tracked based on the received electrical detection signal.

Images