Electrooculography and eye tracking
By arranging compressible conductive foam material electrodes on the earpiece and combining them with signal processing algorithms, the limitations and convenience issues of existing eye-tracking methods have been solved, achieving high-precision gaze direction determination and expanding application scenarios.
Patent Information
- Application Number
- CN202180010758.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-01-19
- Filing Date
- 2021-01-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-01-22
AI Technical Summary
Existing video-based eye-tracking methods require the camera to be pointed at the user's face, which limits the application scenarios, and traditional EOG methods are inconvenient to use in non-medical applications.
A pair of earpieces are used as audio terminals. Each earpiece is equipped with electrodes made of compressible conductive foam material. Voltage signals are obtained by contacting the skin with the electrodes. The gaze direction is determined by combining the signal processing algorithm and the accuracy is improved by using a center voltage estimation and prediction model.
It achieves high-precision eye tracking in non-medical applications, is suitable for a variety of terminal devices, and enhances user experience and the diversity of application scenarios.
Smart Images

Figure BDA0003756975290000101 
Figure HDA0003756975300000011 
Figure HDA0003756975300000012
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 139,022, filed January 19, 2021; U.S. Provisional Patent Application No. 62 / 964,178, filed January 22, 2020; and European Patent Application No. 20153054.0, filed January 22, 2020, all of which are incorporated herein by reference in their entirety. Technical Field
[0003] This invention relates to eye tracking using electrooculography (EOG). In particular, this invention relates to eye tracking using sensors mounted on an earpiece and / or in the temporal region around the ear. Background Technology
[0004] In many cases, eye tracking can be used to understand where a user's attention is focused. Specifically, eye tracking can improve a user's control over peripheral devices.
[0005] The most common eye-tracking method is to acquire video images of the user's eyes. The user's gaze direction can be determined using appropriate image processing and algorithms based on numerical analysis or deep learning. A drawback of this video-based eye-tracking is that the camera must be pointed at the user's face or mounted on their head, which greatly limits its potential applications.
[0006] Recently, alternative methods for video-based eye tracking have been introduced, including electrooculography (EOG). EOG measures the electric dipole potential (the charge difference between the cornea and retina) of the eye's corneal-retinal dipole. As the eye moves along the orbit, the dipole rotates. This potential can be measured using a set of electrodes placed near the orbit and can be used to estimate the eye's position. The accuracy of EOG estimates is approximately 0.5 degrees.
[0007] The standard method for EOG measurement involves placing silver chloride electrodes on both temporal sides around the eye socket. EOG is primarily used for medical applications because using adhesive silver chloride electrodes around the eye is inconvenient.
[0008] However, the use of EOG-based eye tracking has been proposed in other applications. For example, document WO 2018 / 0368722 discloses EOG-based eye tracking, which can be used in applications such as hearing aids. Summary of the Invention
[0009] One object of the present invention is to provide an improved EOG method.
[0010] According to a first aspect of the invention, these and other objectives are achieved by a system for determining a user's gaze direction, the system comprising an audio terminal including a pair of earpieces, each earpiece intended to be worn near a corresponding ear of the user, and a set of electrodes disposed on each earpiece such that they are positioned in use to contact the user's skin, each electrode comprising a patch of compressible and conductive foam material configured to provide a reliable electrical connection (bioelectrode) with the skin. The system also includes circuitry connected to the electrodes, the circuitry being configured to receive a set of voltage signals from the set of electrodes disposed on the audio terminal worn by the user, multiplex the voltage signals to an input signal, remove a predicted center voltage from the input signal to provide a detrending signal, and determine the gaze direction based on the detrending signal. Such compressible and conductive foam material is commonly used for electrical insulation in electronic devices.
[0011] The second aspect of the invention relates to a method corresponding to the first aspect.
[0012] Such conductive foam materials, such as those based on low-density microporous polyurethane foam, offer several advantages, including high conductivity, a soft feel against the skin, satisfactory biosensing performance in a dry state (i.e., no need for conductive paste, for example), and performance that remains consistent across a wide range of compression levels over time. In the case of on-ear headphones, the foam electrodes can be integrated into the earcup with little or no impact on comfort.
[0013] According to some embodiments of the present invention, two estimates are used to determine the center voltage of the next iteration (k+1) using the center voltage of the current iteration (k) and the predicted center voltages of previous iterations k-1, k-2, ..., kN. By using both methods, the accuracy of the center voltage estimation is improved.
[0014] The eye tracking according to the present invention can be used in several applications, including presentation control, headset control, attention decoding, active conversation enhancement, foveal rendering, improved AR / VR experiences, improved content consumption on televisions and mobile devices, drone control, and camera control in filmmaking.
[0015] This invention is compatible with a variety of terminal devices, including but not limited to: headphones, head-mounted displays, hearing aids, smart glasses, ear-worn devices, and earplugs.
[0016] Furthermore, it should be noted that the term "earpiece" should be interpreted broadly and is intended to cover any part of an audio terminal located on the side of the head. For example, in some embodiments of the invention, the earpiece includes a portion of an earplug, an on-ear monitor, a headset, a hearing aid, headphones, or other device (e.g., glasses, AR / VR goggles, etc.).
[0017] According to a third aspect of the invention, these and other objectives are achieved by a system for determining a user's gaze direction, the system comprising a set of electrodes arranged in contact with the user's skin, each electrode comprising a patch of compressible conductive material configured to provide a reliable electrical connection to the skin, and circuitry connected to the electrodes configured to receive a set of voltage signals from the set of electrodes arranged on an audio terminal worn by the user, multiplexing the voltage signals to an input signal, calculating a current center voltage as the difference between the input signal and a voltage representing the current gaze direction, providing a first estimate as a linearly weighted sum of the current center voltage and a currently estimated baseline voltage, providing a second estimate as the current center voltage and the mean of predictions based on a set of previously predicted center voltages, determining a predicted center voltage as the average of the first and second estimates, removing the predicted center voltage from the input signal to provide a detrended signal, and determining the gaze direction based on the detrended signal.
[0018] Accordingly, the electrodes are not disposed on the user-worn terminal device. Instead, the electrodes are attached directly to the user's skin and can be formed by so-called "electronic skin." Signal processing can be similar to or the same as that disclosed in the first aspect of the invention.
[0019] Another aspect of the invention relates to a method corresponding to the third aspect. Attached Figure Description
[0020] The invention will be described in more detail with reference to the accompanying drawings, which illustrate the presently preferred embodiments of the invention.
[0021] Figure 1 The illustration shows a user wearing a pair of headphones according to an embodiment of the present invention.
[0022] Figure 2 exist Figure 2 The image shows the headphone case.
[0023] Figure 3 This is an overview of eye tracking according to embodiments of the present invention.
[0024] Figure 4 Showing more details Figure 3 The eye-tracking block in the middle.
[0025] Figure 5 An overview of the training process according to an embodiment of the present invention is shown.
[0026] Figure 6a This is a diagram showing the measured signal.
[0027] Figure 6b This is a graph showing the signal after baseline removal.
[0028] Figure 6c This is a diagram showing the estimated gaze angle.
[0029] Figure 7 This is a block diagram of a system for determining attention direction according to another embodiment of the present invention.
[0030] Figure 8 This is a block diagram of a system for improving auditory attention according to another embodiment of the present invention.
[0031] Figure 9 This is a block diagram of a system for controlling the pass-through characteristics of noise-canceling headphones according to another embodiment of the present invention. Detailed Implementation
[0032] As an example, Figure 1 and Figure 2 This diagram illustrates how electrodes are positioned on an on-ear headset 1 worn on a user's head 2. "On-ear" refers to the headset 1 comprising two earpieces (left / right) 3 covering the user's ears 4. The edge 5 of each earpiece 3 rests against an area 6 on the user's head 2 surrounding the ears 4. The earpieces 3 are designed to (largely) isolate ambient sound to minimize interference during audio playback. The on-ear headset is designed with a headband 7 that supports the weight of the headset 1 on the head 2. The headband generates horizontal pressure 8, pressing the earpieces 3 against the area 6.
[0033] A set of electrodes 10 is arranged on the terminal to make contact with the user's head during use. The set of electrodes 10 includes multiple lead electrodes 10a and at least one bias electrode 10b. The electrodes 10 can be active or passive.
[0034] like Figure 2 As shown, in the example illustrated, the terminal is implemented using an on-ear earphone 1, with electrodes 10 comprising patches disposed on the edges 5 of each earphone cover 3. In use, horizontal pressure 8 presses the patches 10 against the skin, thereby reducing contact resistance and enhancing biosensing capabilities. The edges 5 have a large contact area 6, thus providing ample usable space for the electrodes to be positioned thereon. The patches 10 can have any suitable shape, such as square, rectangular, circular, or elliptical. In the case of other terminals (e.g., in-ear headphones), different placements and designs of the electrodes 10 may be sufficient.
[0035] Patch 10 can be manufactured using a conductive foam with high conductivity. Preferably, the material is capable of maintaining its thickness at temperature and providing a consistent and reliable electrical connection. The electrode material may have a low vertical resistance (e.g., less than 0.005 Ω) and the ability to maintain optimal conductivity during compression.
[0036] For example, the material used for the electrode can have a thickness of 0.33 to 0.53 mm and a vertical contact resistance of 0.001 to 0.003 Ω. The volume resistivity can be typically 1.3 to 2.94 Ω·cm. The material can have a surface resistivity of 0.1 to 0.12 Ω / sq. When a 0.12 N / mm² is applied... 2 Under stress, the resistance changes from 0.09Ω to 0.001Ω, therefore the small change in resistivity is related to the compression of the material. The electrodes can be manufactured with viscous sides, which allows for easy integration with plastic or metal housings.
[0037] For example, the electrodes can be made by Rogers' Condux Plus. TM Made of foam.
[0038] Note that materials other than foam can also be used for electrode patches. For example, fabric patches or other conductive materials can be used for electrodes.
[0039] Sensor materials at even thinner nanometer scales can also be considered. These thin, stretchable electrodes can be directly attached to the user's skin and are sometimes referred to as "electronic skin." Electronic skin can be fabricated by combining oxide materials with biocompatible silicon and can include self-modified coatings, such as those made of vanadium dioxide. Further details are provided by Rehman et al. in their September 1, 2020 article, *Artificial Somatosensors: Feedback Receptors for ElectronicSkins*.
[0040] Now turn to the estimation of eye fixation, and refer to... Figure 3 The signal from lead electrode 10a is modulated in the analog domain using instrumentation amplifier 11, and the signal from bias electrode 10b is used in the process to reduce common-mode noise. The modulated signal is multiplexed, converted by A / D converter 12, and provided to eye-tracking processing circuit 13, which identifies changes in gaze direction and determines the instantaneous gaze direction (e.g., relative gaze angle or direction). Finally, the output from processing circuit 13 is independently used to control some facility, application, or peripheral device 14. Examples of applications in which eye tracking according to the invention can be used include presentation control, headset control, active dialogue enhancement, foveal rendering, improved AR / VR experiences, improved content consumption on televisions and mobile devices, drone control, and camera control in filmmaking.
[0041] Figure 6aAn example of the signal input from A / D converter 12 to circuit 13 is shown. The gray area 51 represents the actual signal, which is noisy and fluctuates around the average value. The dashed line 52 is the voltage representing the forward (center) gaze direction (called the "center voltage"). The two dotted lines 53 are the left and right voltage limits. These voltages represent extreme left or right gazes and are separated from the center voltage 52 by a fixed δ voltage, which is constant for each individual.
[0042] Continue to refer to Figure 6a The amplitude (voltage) of the measurement signal 51 is related to the user's gaze direction. The difference 57 between the current measurement voltage 51 and the center voltage 52 is called gaze ΔV and represents the current gaze angle (e.g., gaze direction; gaze direction relative to a reference direction or angle). A person's gaze shifts between focal points, a process called saccades. The transition between saccades is typically characterized by the eye moving relatively quickly or jerkily from one focal point to another.
[0043] The center voltage 52 is affected by a non-negligible drift. Figure 6a (The downward tilt in the middle). For a specific saccade, i.e., a constant gaze ΔV 57, the measured voltage will therefore also drift. The difference between the measured signal and the (drifted) center voltage always represents the voltage 57 in the current gaze direction.
[0044] By predicting the drift of the center voltage 52, the gaze estimator 13 is able to detect a shift to another gaze direction, such as a new saccade. In the following text, the predicted value of the center voltage for iteration k is referred to as the baseline (k).
[0045] Eye tracking processing circuit 13 in Figure 4 The details are shown in more detail below and will be discussed further. Figure 4 The processing shown will be able to handle gaze directions within a plane, such as horizontal (left and right) and vertical (up and down). Therefore, two parallel processes are actually needed, one for each plane. For simplicity, only one (left-right) will be discussed here.
[0046] Another type of eye movement is vergence, which is related to the depth of focus. In this case, the eyes move in opposite directions, either inward or outward. Vergence is associated with applications involving distance variations ranging from less than 1 meter to more than 3 meters. Interactions with mobile phones and other objects typically involve vergence variations. Watching a regular television does not involve vergence variations, but watching an automated stereoscopic television with depth display may.
[0047] In the currently disclosed embodiments, the algorithm operates on a data window (segment) comprising a set of samples. The window length is a variable parameter and does not affect the principle of the algorithm. For example, the sampling rate could be 250 Hz, and the window length could be 1 / 10 of a second, i.e., 25 samples.
[0048] Each window (index k) of the signal is first processed in block 21 to remove power line noise. In this example, this is achieved using a moving average filter. Alternatively, a digital IIR or FIR filter can be used for the same purpose. The output of block 21 is the filtered signal (index k).
[0049] Next, in block 22, the baseline (k), such as the estimated center voltage for the current window, is removed. The estimation of the center voltage will be described below with reference to blocks 31-36. Note that the baseline (k) is a single value representing the baseline prediction over the entire next window (i.e., 25 data points in this case).
[0050] For baseline removal, the value of the baseline (k) is subtracted from each sample in the filtered signal (k) according to the following formula:
[0051] Detrended signal(k) = Filtered signal(k) - Estimated baseline(k)
[0052] Following baseline removal, the detrended signal is used in gaze estimator 23 to estimate the angle of horizontal gaze. In the current implementation, this is obtained using a linear regression model calibrated with the maximum left gaze and maximum right gaze voltage as upper and lower limits, respectively. The maximum left and maximum right voltages correspond to the voltages of the maximum possible left and right gaze angles and depend on the center voltage, which will be described below in reference blocks 30a and 30b. Other alternatives regarding how to estimate gaze based on the detrended signal without using a linear regression model may also be possible.
[0053] The detrended signal is also used in state classifier 24 for state classification, such as determining whether the horizontal gaze direction is left-handed, right-handed, or centered. This is achieved using a linear state classifier. In this implementation, linear discriminant analysis is used. Alternatively, other multi-class classifiers (such as logistic regression or support vector machines) can be used. The same task can also be accomplished using neural networks. Although only three states are included in this example, more states can be added to the classification (e.g., intermediate states such as center-left or center-right, and multiple states suitable for a resolution that can achieve SNR, i.e., the number of possible positions consistent with the aforementioned 0.5-degree accuracy).
[0054] As mentioned above, the outputs from block 23 and block 24 can be used independently to drive application 14.
[0055] The detrending signal is also provided to the transition classifier 25 to identify transitions, i.e., changes in gaze caused, such as from one saccade to another or by “smoothly following” eye movements. Each time a new saccade occurs, the detrending signal will show a vertical voltage shift (up or down). In other words, the difference between the measured voltage and the central voltage (direct gaze) changes as the user moves their eyes. The transition classifier 25 processes each data window (index k) to determine if it includes such a voltage shift. Voltage shifts representing transitions to new saccades are typically rapid and can be identified in the detrending signal as steps or ramps over short time intervals. The transition classifier 25 can also identify shifts caused by “smoothly following” eye movements.
[0056] If the current window does not contain any identifiable shifts (meaning the user is not moving their eyes), then the gaze ΔV will remain constant according to the following formula:
[0057] Fixation ΔV(k) = Fixation ΔV(k-1)
[0058] In addition, the center voltage calculator 27 uses the observed ΔV(k) to calculate the current center voltage (k) according to the following formula:
[0059] Center voltage (k) = Median [Filtered signal (k) - Eye contact ΔV (k)]
[0060] Here, a single value gaze ΔV(k) is subtracted from each sample of the filtered signal (k). Note that the center voltage (k) is not predicted in the same way as the baseline (k), but is based on the perception of the signal (and the determined gaze) measured during the window (k).
[0061] If a transition is detected by transition classifier 25, the timestamp of that transition (which sample) is provided to block 28, where a new gaze ΔV is determined. Block 28 also receives the filtered signal (k) (i.e., before detrending) and the baseline (k). The new gaze ΔV can then be determined based on the voltage level of the filtered signal after that timestamp, according to the following formula:
[0062] Focusing on ΔV(k) = taking the median of the filtered signal(k) 转变后 -baseline(k)]
[0063] The new gaze ΔV value is stored and used in subsequent iterations until another transition is detected. Furthermore, the center voltage estimator 29 estimates the current center voltage as...
[0064] Center voltage (k) = baseline (k).
[0065] The left voltage estimator 30a and the right voltage estimator 30b use the value of the center voltage (k) to determine the new potential maximum voltage for the measured voltage. Note that this maximum / minimum voltage is not required for gaze estimation. However, in some implementations, they can be used to improve the accuracy of the gaze estimator 23 and the state classifier 24. By setting these thresholds to appropriate values, so-called "smooth following" of eye movements can also be achieved.
[0066] Note that the gaze direction estimation (block 23) and state classification (block 24) are described as being performed for each iteration. Therefore, the left and right maximum voltages are also determined in each iteration (blocks 30a, 30b). This may improve reliability but reduce computational efficiency. If the transition detection in block 25 is reliable, it may be advantageous to estimate the gaze and perform state classification only when a transition is detected in block 25. In this case, blocks 23 and 24 would immediately follow block 28.
[0067] The prediction of the center voltage for the next iteration, referred to as the baseline, will now be described with reference to blocks 31-36.
[0068] The first estimated baseline A is generated using the leakage integrator 31, forming a linear weighted sum of the previous center voltage and the previous estimated baseline:
[0069] Baseline A (k+1)=α·center voltage(k)+(1-α)·baseline(k)
[0070] The leakage integrator 31 can be replaced by a bandpass filter with adjustable low and high cutoff frequencies for smooth tracking.
[0071] The second estimated baseline B is generated using register 32 and AR model 33. Register 32 stores a set of N+1 prior predicted baselines (k), baseline (k-1), ..., baseline (kN), and this set is provided to AR model 33. AR model 33 generates the predicted baseline. AR模型 (k+1), and then averaged with the center voltage (k) in block 35 to provide a second estimate B, as follows:
[0072]
[0073] The two estimated baselines A and B are averaged in block 36 to obtain the final baseline (k+1), which will be stored in register 32 and used in block 22 in subsequent iterations to detrend the filtered signal.
[0074] Figure 4The process requires a short training session to function properly. During the training session, the user is asked to perform several eye "gestures" (moving the eyes from right to center, from center to left, from left to center, and from center to right). Additional gestures can be added to improve accuracy or to increase the number of states that need to be recognized (as mentioned earlier, it is also possible to classify states such as center-left or center-right). In applications explicitly linked to a display, for greater precision, the user can be asked to fix the target on the display at a known location.
[0075] The training process is used to train the models (estimation model, classification model, and transformation model) used in blocks 23, 24, and 25. Figure 5 In one embodiment shown, a linear regression model is used for classification and gaze estimation. The training process needs to be performed once for each individual and is specific to each individual, and training data 41 can be collected using a simple set of eye poses. The collected data 41 is used to train a first model 42, which is used in a state classifier (block 24) to classify gaze directions in possible states (center-right-left-up-down). Furthermore, the collected data 41 is used to train a second model 43, which is used in a transition detector 25 to identify transitions from one saccade to another. Finally, the collected data 41 is used to train a third model 44, which is used in a gaze estimator 23 to estimate horizontal and vertical gaze.
[0076] Note that the system disclosed in this paper can, in principle, work without training, by simply choosing approximate parameter values (at the cost of lower accuracy).
[0077] Figure 6b The signal 54 is shown (e.g., the signal input 51 after signal processing and baseline removal in blocks 21 and 22, which is...). Figure 4 (The signal inputs of the two branches). Clearly, Figure 6a The visible baseline drift has now been eliminated. Classification boundaries (e.g., those related to maximum / minimum voltage) are indicated by dashed line 55.
[0078] Figure 6c It shows the result of Figure 6b The estimated gaze angle value 56 is generated by signal 54 in the signal.
[0079] The EOG method for eye fixation detection described above can be advantageously combined with additional sensors or measurement units, particularly with additional sensors for measuring physiological responses. For example... Figure 7 As shown, in addition to EOG electrodes 10a and 10b, system 70 may include at least one of the following additional sensor systems:
[0080] Inertial Measurement Unit (IMU) 71, used to measure acceleration, magnetic field and angular velocity.
[0081] Electrodes 72 used for measuring electroencephalogram (EEG)
[0082] Electrode 73 for measuring electromyography (EMG)
[0083] Sensor 74 for measuring heart rate and blood pressure (photoplethysmography, PPM)
[0084] Temperature sensor 75
[0085] A variety of sensors can be envisioned, including display-based sensors, visible wavelength camera sensors, simultaneous positioning and mapping sensors, thermal imagers, head-mounted display sensors, in-ear sensors, wrist sensors, gaze position sensors, pupil diameter sensors, facial expression sensors, head position sensors, viewing distance sensors, facial expression sensors, valence state sensors, wake-up sensors, electroencephalogram (EEG) sensors, specifically positioned electrodes, thermal sensors, optical sensors, electrooculogram (EOG) sensors, respiration sensors, heart rate sensors based on plethysmography, skin conductance sensors, gas sensors, CO2 content sensors, R3COH content sensors, or seat-based sensors.
[0086] The data collected from all these sensors and systems is provided to one or more devices for appropriate preprocessing. For example, see the reference above. Figure 4 The processing discussed can be performed on the EOG signal. Depending on the situation, the device may be included in an ear-hook headset 76, an in-ear headset 77, or an on-ear headset 78.
[0087] An appropriately trained neural network 79 is connected to receive preprocessed signals and estimate the direction of attention and / or eye gaze.
[0088] The system according to the present invention can have a variety of different applications.
[0089] Figure 8 A block diagram of system 80 is shown, which is configured to improve the brain's ability to focus an individual's auditory attention (the effect of selective attention in the brain) on a specific stimulus while filtering out a range of other stimuli. This focus allows a person to concentrate on a single conversation in a noisy room (such as at a party), a phenomenon sometimes referred to as the "cocktail party effect." In some cases, the natural brain's ability to pick out a particular stream of conversation is insufficient to track a conversation without cognitive load. Figure 8 In which situations can the system enhance the dialogue?
[0090] Sensor systems (e.g.) Figure 7The system 70 shown receives biophysical data and provides the user's attention direction. In parallel, a mixture 81 of dialogue and noise is collected by a microphone array, preferably comprising beamforming microphones. A blind source separation processor 82 receives the collected mixture 81 and attention direction, and is configured to extract relevant dialogue from the mixture 81 measured by the microphone array. Noise can be attenuated or eliminated by an attenuator 83. The relevant dialogue is then amplified by an amplifier 84 and output to the user through a suitable speaker 85.
[0091] Figure 9 A block diagram of system 90 is shown, which is configured to control the transmission characteristics of noise-canceling headphones. Modern noise-canceling headphones typically have the ability to allow audio transmission when a user is engaged in conversation or when conversation is received from the surroundings. Figure 8 The system described in the paper introduces the possibility of controlling the directionality of audio transmission using biophysical sensors.
[0092] Sensor systems (e.g.) Figure 7 The system 70 shown receives biophysical data and provides the user's attention direction. In parallel, conversations or sounds of interest 91 are collected by a microphone array, which preferably includes beamforming microphones. In the blind source separation processor 92, similar... Figure 8 The algorithm described uses attention orientation to separate the collected mix into noise and relevant audio, such as audio from the user's attention orientation or the dialogue or sounds the user is engaging with. Noise can be eliminated by attenuator 93. Relevant audio is amplified by amplifier 94 and provided to mixer 95, where it is mixed with the content currently received by the headphones and intended for playback. Therefore, this mix is played back by the headphone speakers 96.
[0093] Conclusion
[0094] As used herein, unless otherwise stated, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe a common object merely indicates that different instances of the same object are being referenced, and does not mean that the objects described in this way must be in a given order in time, space, sequence, or any other way.
[0095] In the claims below and in the description herein, the term "comprising" is an open term, meaning that it includes at least the elements / features listed thereafter, but does not exclude other elements / features. Therefore, when used in the claims, the term "comprising" should not be construed as limiting itself to the means, elements, or steps listed thereafter. For example, the scope of the statement "a device comprising A and B" should not be limited to the device consisting only of elements A and B. The term "comprising" as used herein is also an open term, meaning that it includes at least the elements / features listed thereafter, but does not exclude other elements / features. Therefore, "comprising" is synonymous with "including".
[0096] As used herein, the term "exemplary" is used in the sense of providing an example, as opposed to indicating quality. That is, an "exemplary embodiment" is an embodiment provided as an example, and not necessarily an embodiment of exemplary quality.
[0097] It should be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes combined in a single embodiment, drawing, or description thereof for the purpose of simplifying the disclosure and aiding in the understanding of one or more aspects of the invention. However, the method of this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspect is less than all the features of a single foregoingly disclosed embodiment. Therefore, the claims following the detailed description are thus expressly incorporated into that detailed description, each claim serving as a separate embodiment of the invention.
[0098] Furthermore, although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments should be within the scope of the invention and form different embodiments, as will be understood by those skilled in the art. For example, in the following claims, any claimed embodiment may be used in any combination.
[0099] Furthermore, some embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the function. Thus, a processor having the necessary instructions for performing such methods or method elements forms a means for performing such methods or method elements. Furthermore, the elements of the apparatus embodiments described herein are examples of means for performing functions performed by such elements for the purposes of achieving the present invention.
[0100] Numerous specific details are set forth in the description provided herein. However, it should be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0101] Similarly, it should be noted that the term "coupling," when used in the claims, should not be construed as limited to direct connections. The terms "coupling" and "connection," as well as their derivatives, may be used. It should be understood that these terms are not intended to be synonyms with each other. Therefore, the scope of stating that device A is coupled to device B should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. This implies the existence of a path between the output of A and the input of B, which may include other devices or apparatus. "Coupling" can mean two or more elements in direct physical or electrical contact, or two or more elements not in direct contact with each other but still cooperating or interacting with each other.
[0102] Therefore, although specific embodiments of the invention have been described, those skilled in the art will recognize that other and further modifications can be made without departing from the spirit of the invention, and all such changes and modifications fall within the scope of the invention. For example, any formulas given above are merely representative of programs that can be used. Functions can be added to and removed from the block diagram, and operations can be interchanged between functional blocks. Within the scope of the invention, steps can be added to or removed from the described methods. For example, in the illustrated embodiment, the terminal device is illustrated as a pair of on-ear headphones. However, the invention is also applicable to other terminal devices, such as in-ear headphones and hearing aids.
Claims
1. A system for determining a user's gaze direction, comprising: An audio terminal includes a pair of earpieces, each earpiece intended to be worn near the user's corresponding ear. A set of electrodes is arranged on each earpiece such that the set of electrodes is placed in contact with the user's skin during use. Each electrode includes a patch of compressible and conductive foam material configured to provide a reliable electrical connection to the skin. A circuit connected to the electrodes, the circuit being configured as follows: A set of voltage signals is received from a set of electrodes placed on the user's worn audio terminal. The voltage signal is multiplexed into the input signal. The predicted center voltage representing the forward gaze direction is removed from the input signal to provide a detrending signal, and The gaze direction is determined based on the detrending signal. in, The predicted center voltage is determined as follows: Calculate the previous center voltage as the difference between the input signal and the voltage representing the previous gaze direction. A first estimate is provided as a linear weighted sum of the previous center voltage and the previously predicted center voltage. A second estimate is provided as the average of the previous center voltage and the predictions based on a set of prior predicted center voltages, and The average of the first estimate and the second estimate is determined as the predicted center voltage.
2. The system according to claim 1, wherein, The audio terminal is a pair of on-ear headphones, each of which includes an earpiece configured to rest against the user's scalp, and wherein the electrodes are arranged on the edge of the earpiece for resting against the skin.
3. The system according to claim 1, wherein, The thickness of the conductive foam material is less than 1 mm, and the vertical contact resistance is less than 0.005 Ω.
4. The system according to claim 1, wherein, The volume resistivity of the conductive foam material is in the range of 1.3 to 3 Ω·cm.
5. The system according to claim 1, wherein, The conductive foam material has a surface resistivity of 0.1 to 0.12 Ω / sq.
6. The system according to any one of claims 1-5, wherein, The predicted center voltage is the current predicted center voltage, and the determined gaze direction is the current gaze direction. The circuit is further configured to determine the subsequent predicted center voltage by: Calculate the current center voltage as the difference between the input signal and the voltage representing the current gaze direction. Provides a first estimate as a linear weighted average of the current center voltage and the currently predicted center voltage. Provides a second estimate as the current center voltage and the average of predictions based on a set of prior predicted center voltages, and The subsequent predicted center voltage is determined as the average of the first estimate and the second estimate. The circuit is also configured to determine the subsequent gaze direction based on the determined subsequent prediction center voltage.
7. The system according to claim 6, wherein, The predictions were made using an AR model and the set of predicted center voltages.
8. The system according to any one of claims 1-5, wherein the circuitry is further configured to identify an offset in the de-trending signal, the offset representing a shift in the user's gaze from one saccade to another, and The circuitry is also configured to determine the subsequent gaze direction based on the identified offset in the detrending signal.
9. The system according to any one of claims 1-5, wherein the circuit is further configured to determine a state of the gaze direction, the state being selected from the group consisting of a left state and a right state, wherein the left state and the right state are defined by a detrending signal exceeding or falling below a defined threshold.
10. A method for determining a user's gaze direction, comprising: An audio terminal is provided, comprising a pair of earpieces, each earpiece supporting a set of electrodes, the electrodes comprising patches of compressible and conductive material configured to provide a reliable electrical connection with a user's skin. The audio terminal is placed on the user so that the electrodes are in contact with the user's skin near the corresponding ears. A set of voltage signals is obtained from a set of electrodes placed on the user's worn audio terminal. The voltage signal is multiplexed into the input signal. The predicted center voltage representing the forward gaze direction is removed from the input signal to provide a detrending signal, and The gaze direction is determined based on the detrending signal. in, The predicted center voltage is determined as follows: Calculate the previous center voltage as the difference between the input signal and the voltage representing the previous gaze direction. A first estimate is provided as a linear weighted sum of the previous center voltage and the previously predicted center voltage. A second estimate is provided as the average of the previous center voltage and the predictions based on a set of prior predicted center voltages, and The average of the first estimate and the second estimate is determined as the predicted center voltage.
11. The method according to claim 10, wherein, The predicted center voltage is the current predicted center voltage, and the determined gaze direction is the current gaze direction. The method further includes determining the subsequent predicted center voltage as follows: Calculate the current center voltage as the difference between the input signal and the voltage representing the current gaze direction. Provides a first estimate as a linear weighted average of the current center voltage and the currently predicted center voltage. Provides a second estimate as the current center voltage and the average of predictions based on a set of prior predicted center voltages, and The subsequent predicted center voltage is determined as the average of the first estimate and the second estimate. The method further includes determining the subsequent gaze direction based on the determined subsequent predicted center voltage.
12. The method according to claim 11, wherein, The predictions were made using an AR model and the set of predicted center voltages.
13. The method of claim 10, further comprising identifying a step change in the detrending signal, the step change representing a shift in the user's gaze from one saccade to another, and The method further includes determining the subsequent gaze direction based on the identified step changes in the detrending signal.
14. The method of claim 10, further comprising determining a state of the gaze direction, the state being selected from the group consisting of a left state and a right state, wherein the left state and the right state are defined by a detrending signal exceeding or falling below a defined threshold.
15. The method according to any one of claims 10-14, wherein, The audio terminal is a pair of on-ear headphones, each of which includes an earpiece configured to rest against the user's scalp, and wherein the electrodes are arranged on the edge of the earpiece intended to rest against the skin.
16. A system for determining the direction of attention, comprising: The system for determining a user's gaze direction according to any one of claims 1-9, At least one additional sensor, A neural network is used to estimate the attention direction given inputs from the system and from the at least one additional sensor.
17. The system of claim 16, wherein the at least one additional sensor comprises at least one sensor for measuring physiological responses.
18. The system of claim 16, wherein the additional sensor comprises at least one of an inertial measurement unit (IMU), a sensor for measuring heart rate, a sensor for measuring blood pressure, a temperature sensor, and electrodes for electroencephalography (EEG), electromyography (EEG), and / or electromyography (EMG).
19. A system for improving auditory attention, comprising: The system for determining attention direction according to any one of claims 16-18, A blind source separation algorithm is connected to receive the attention direction and a mix including dialogue and noise. The blind source separation algorithm is configured to extract the dialogue received from the attention direction. An amplifier is used to amplify the extracted dialogue.
20. A system for controlling the transmission characteristics of noise-canceling headphones, the system comprising: The system for determining attention direction according to any one of claims 16-18, A blind source separation algorithm is connected to receive the attention direction and a mix including dialogue and noise. The blind source separation algorithm is configured to extract the dialogue received from the attention direction. An amplifier, used to amplify the extracted dialogue, and A mixer is used to blend the extracted dialogue into content intended to be played back on the headphones.
21. A method for determining a user's gaze direction, comprising: A set of electrodes is arranged to contact the user's skin. Each electrode includes a patch of compressible and conductive material configured to provide a reliable electrical connection with the skin. A set of voltage signals is received from a set of electrodes placed on the user's worn audio terminal. The voltage signal is multiplexed into an input signal. Calculate the center voltage, which is the difference between the input signal and the voltage representing the previous gaze direction. A first estimate is provided as a linear weighted sum of the center voltage and the previously estimated baseline voltage. Provides a second estimate as the center voltage and the average of a set of prior predicted center voltages, and The predicted center voltage is determined as the average of the first and second estimates. The predicted center voltage is removed from the input signal to provide a detrending signal, and The gaze direction is determined based on the detrending signal.
22. A system for determining a user's gaze direction, comprising: A set of electrodes is arranged to contact the user's skin, each electrode including a patch of compressible conductive material configured to provide a reliable electrical connection with the skin. A circuit, connected to the electrode, is configured to: A set of voltage signals is received from a set of electrodes placed on the user's worn audio terminal. The voltage signal is multiplexed into an input signal. Calculate the center voltage, which is the difference between the input signal and the voltage representing the previous gaze direction. A first estimate is provided as a linear weighted sum of the center voltage and the previously estimated baseline voltage. A second estimate is provided as the center voltage and the mean of a set of prior predicted center voltages. The predicted center voltage is determined as the average of the first and second estimates. The predicted center voltage is removed from the input signal to provide a detrending signal, and The gaze direction is determined based on the detrending signal.
23. A method for improving auditory attention, comprising: Receives the attention direction determined by the system according to any one of claims 16-18, and a mix including dialogue and noise. Extract the dialogue received from the attention direction from the mix, and Zoom in on the extracted dialogue.
24. A method for controlling the transmission characteristics of noise-canceling headphones, the method comprising: Receives the attention direction determined by the system according to any one of claims 16-18, and a mix including dialogue and noise. Extract the dialogue received from the attention direction from the audio mix. Zoom in on the extracted dialogue, and The amplified, extracted dialogue is blended into the content intended to be played back on the headphones.
25. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 10 to 15, 21, and 23 to 24.
26. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 10 to 15, 21, and 23 to 24.
27. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 10 to 15, 21, and 23 to 24.
28. An apparatus comprising components for performing the method according to any one of claims 10 to 15, 21 and 23 to 24.
Citation Information
Patent Citations
Eye-gaze tracking device, eye-gaze tracking method, electro-oculography measuring device, wearable camera, head-mounted display, electronic eyeglasses, and ophthalmological diagnosis device
US20110170067A1
Smart audio headphone system
US20170339484A1
