Eye movement tracking method based on high-frequency sound waves

Abstract

The invention relates to eye movement tracking, in particular to an eye movement tracking method based on high-frequency sound waves, which comprises the following steps: a sound wave emitter emits high-frequency sound waves to an eye region; the sound wave receiver receives echo signals formed by reflection of eyeballs and surrounding tissues; the signal processing module carries out signal processing on the echo signal and generates an echo profile according to the processed echo signal; inputting the echo profile into a pre-trained eye movement tracking model, extracting features from the echo profile by the eye movement tracking model, and predicting coordinates of a fixation point; according to the technical scheme provided by the invention, the defect that accurate eye movement tracking is difficult to realize on the premise of effectively protecting user privacy in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to eye tracking, and in particular to an eye tracking method based on high-frequency sound waves. Background Art

[0002] In the fields of human-computer interaction, psychology research, medical diagnosis, and virtual reality (VR) / augmented reality (AR), eye tracking, as a key technology, is of great significance for understanding user intentions, attention, emotional state, cognitive processes, and improving user experience. Traditional eye tracking methods mainly rely on optical principles, such as using infrared cameras to capture changes in reflected light spots on the surface of the eyeball, or pupil-corneal reflection (PCR) based on video image processing. However, existing eye tracking methods face many challenges in practical applications:

[0003] 1) Ambient light interference: Optical eye tracking systems are extremely sensitive to ambient light. Under strong or complex lighting conditions, the recognition accuracy of reflected light points will be greatly reduced, resulting in inaccurate tracking results.

[0004] 2) Wearing comfort and privacy risks: For devices that need to be worn for extended periods of time (such as VR glasses), optical sensors significantly increase the size and weight of the device, reducing user comfort. Furthermore, the use of cameras raises concerns about privacy leaks.

[0005] 3) Head movement limitations: Existing eye tracking methods require the user's head to remain relatively still, limiting the user's freedom of movement. This is particularly inconvenient in certain application scenarios (such as sports analysis and outdoor activities).

[0006] 4) Cost and technical barriers: The R&D and production costs of high-precision optical eye tracking systems are high, and the hardware performance requirements are high, which limits their widespread application;

[0007] 5) Applicability to special populations: For people who wear glasses, contact lenses, or have eye diseases, the accuracy of optical eye tracking will be affected to a certain extent. Summary of the Invention

[0008] (1) Technical problems solved

[0009] In response to the above-mentioned shortcomings of the existing technology, the present invention provides an eye tracking method based on high-frequency sound waves, which can effectively overcome the defects of the existing technology that it is difficult to achieve accurate eye tracking while effectively protecting user privacy.

[0010] (2) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0012] An eye tracking method based on high-frequency sound waves comprises the following steps:

[0013] S1, the sound wave transmitter emits high-frequency sound waves to the eye area;

[0014] S2, the sound wave receiver receives the echo signal formed by the reflection of the eyeball and surrounding tissues;

[0015] S3, the signal processing module processes the echo signal and generates an echo profile according to the processed echo signal;

[0016] S4. Input the echo profile into the pre-trained eye tracking model. The eye tracking model extracts features from the echo profile and predicts the gaze point coordinates.

[0017] Preferably, the signal processing module in S3 processes the echo signal and generates an echo profile according to the processed echo signal, including:

[0018] S31, performing band-pass filtering on the echo signal;

[0019] S32, preprocessing the echo signal after bandpass filtering;

[0020] S33. Perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal, and convert the cross-correlation calculation result into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile.

[0021] Preferably, performing bandpass filtering on the echo signal in S31 includes:

[0022] According to the frequency characteristics of the sound wave transmitter, the upper and lower cutoff frequencies of the bandpass filter are set to ensure that only the echo signal that matches the frequency of the transmitted high-frequency sound wave is retained, and the digitized echo signal is bandpass filtered using the bandpass filter.

[0023] Preferably, in S32, preprocessing the echo signal after bandpass filtering includes:

[0024] S321, aligning the bandpass filtered echo signal with the transmitted high-frequency sound wave, and normalizing them to the same amplitude range;

[0025] S322: Perform windowing processing on the normalized echo signal to reduce spectrum leakage.

[0026] Preferably, in S33, cross-correlation calculation is performed on the transmitted high-frequency sound wave and the pre-processed echo signal, and the cross-correlation calculation result is converted into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile, including:

[0027] S331. Directly perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal:

[0028]

[0029] Where R(τ) is the cross-correlation function value, which indicates the correlation between the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n] under the time shift parameter τ. τ is the time shift parameter, which indicates the time offset of the preprocessed echo signal y[n] relative to the transmitted high-frequency sound wave x[n]. n is a discrete time variable, which indicates the sampling point index of the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n]. N is the signal length, which indicates the total number of sampling points in the signal.

[0030] Or use FFT to accelerate the cross-correlation calculation between the transmitted high-frequency sound wave and the pre-processed echo signal:

[0031] R(τ)=IFFT(FFT(x[n])·conj(FFT(y[n])));

[0032] Where FFT(·) represents fast Fourier transform, conj(·) represents complex conjugate, and IFFT(·) represents inverse fast Fourier transform.

[0033] S332. Determine the time shift parameter corresponding to the maximum cross-correlation function value in the cross-correlation calculation result. If multiple maximum cross-correlation function values exist, determine a reasonable maximum cross-correlation function value based on prior knowledge.

[0034] S333. Calculate the distance between the acoustic wave receiver and the eyeball based on the time shift parameter corresponding to the maximum cross-correlation function value:

[0035]

[0036] Where d is the distance between the sound wave receiver and the eyeball, τ' is the time shift parameter corresponding to the maximum cross-correlation function value, and c is the speed of sound;

[0037] S334. According to the distance between the sound wave receiver and the eyeball and the position information of the sound wave receiver, a three-dimensional profile reflecting the position of the eyeball is determined by using triangulation or least squares method to generate an echo profile.

[0038] Preferably, in S4, the echo profile is input into a pre-trained eye tracking model, and the eye tracking model extracts features from the echo profile and predicts the gaze point coordinates, including:

[0039] S41, performing model training on the eye tracking model to obtain a pre-trained eye tracking model;

[0040] S42, preprocessing the echo profile, and inputting the preprocessed echo profile into a pre-trained eye tracking model;

[0041] S43. Utilize the residual block in the eye tracking model to extract deep features of the echo profile. The residual block alleviates the gradient vanishing problem through skip connections and improves feature extraction capabilities.

[0042] S44. Using the fully connected layer in the eye tracking model, the extracted deep features are mapped to the gaze point coordinates in the screen coordinate system or the three-dimensional space coordinate system;

[0043] Among them, the eye tracking model is built based on the residual network ResNet-18.

[0044] Preferably, the eye tracking model is trained in S41 to obtain a pre-trained eye tracking model, including:

[0045] S411, collecting echo profiles under different gaze directions, eye movement states, and head postures, as well as corresponding screen indicator point coordinates, annotating the echo profiles using the screen indicator point coordinates, and constructing a training data set;

[0046] S412, dividing the training data set into a training set, a validation set, and a test set according to a preset ratio;

[0047] S413, setting the loss function and optimizer of the eye tracking model;

[0048] S414, inputting the training set into the eye tracking model for model training;

[0049] S415, calculating the loss value based on the loss function, and the optimizer updating the model parameters according to the loss value and the network gradient information;

[0050] S416. If the loss value is less than the preset threshold, the model training ends and the current eye tracking model is the trained eye tracking model. Otherwise, the process returns to S414 and continues to train the model using the training set.

[0051] S417. Input the validation set into the trained eye tracking model, evaluate the generalization ability of the model by observing its performance on the validation set, and tune the model's hyperparameters and structure;

[0052] S418. Input the test set into the tuned eye tracking model to evaluate the model performance.

[0053] Preferably, the sound wave transmitter includes a first speaker and a second speaker symmetrically embedded in two glasses legs, the first speaker transmits 18-21kHz FMCW sound waves to the eye area, and the second speaker transmits 21.5-24.5kHz FMCW sound waves to the eye area.

[0054] Preferably, the sound wave receiver includes 8 miniature microphones, and the 8 miniature microphones are symmetrically embedded in two glasses legs.

[0055] Preferably, the signal processing module and the eye tracking model are integrated into a processing unit, and the processing unit adopts Teensy 4.1 or MAX78002 as a microcontroller and integrates a low-power CNN accelerator.

[0056] (3) Beneficial effects

[0057] Compared with the existing technology, the eye tracking method based on high-frequency sound waves provided by the present invention has the following beneficial effects:

[0058] 1) By emitting high-frequency sound waves to the eye area for eye tracking, sensitivity to ambient light is significantly reduced, preventing interference from ambient light. Accurate eye tracking can be performed in strong or complex lighting conditions. It can also accurately track eye movements for people who wear glasses, contact lenses, or have eye diseases.

[0059] 2) The overall structure is lightweight (approximately 44.2g), making it easy for users to wear the device for extended periods of time. It also eliminates the need to restrict head movement during use, effectively improving wearing comfort.

[0060] 3) No need to collect facial images, which will not cause user privacy leakage;

[0061] 4) Low power consumption (≤95.4mW), significantly improving battery life;

[0062] 5) The R&D and production costs are low, and the hardware performance requirements are not high. It is compatible with a variety of glasses frames and can be integrated into commercial glasses, which will help it to be widely used in medical, automotive and virtual reality (VR) / augmented reality (AR) fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0064] Figure 1 It is a schematic diagram of the process of the present invention;

[0065] Figure 2 Schematic diagram of the workflow of the signal processing module in the present invention;

[0066] Figure 3This is a flow chart of the eye tracking model in the present invention predicting the coordinates of the gaze point based on the echo profile. DETAILED DESCRIPTION

[0067] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0068] An eye tracking method based on high-frequency sound waves, such as Figure 1 As shown, S1, the sound wave transmitter emits high-frequency sound waves to the eye area.

[0069] In the technical solution of the present application, the sound wave transmitter includes a first speaker and a second speaker symmetrically embedded in two glasses legs (one on each side). The first speaker transmits FMCW sound waves of 18 to 21 kHz to the eye area, and the second speaker transmits FMCW sound waves of 21.5 to 24.5 kHz to the eye area.

[0070] S2. The sound wave receiver receives the echo signal formed by the reflection of the eyeball and surrounding tissues.

[0071] In the technical solution of the present application, the sound wave receiver includes 8 miniature microphones, and the 8 miniature microphones are symmetrically embedded in two glasses legs (four on each side).

[0072] S3, the signal processing module processes the echo signal and generates an echo profile based on the processed echo signal, such as Figure 2 As shown, specifically including:

[0073] S31, performing band-pass filtering on the echo signal;

[0074] S32, preprocessing the echo signal after bandpass filtering;

[0075] S33. Perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal, and convert the cross-correlation calculation result into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile.

[0076] Specifically, performing bandpass filtering on the echo signal in S31 includes:

[0077] According to the frequency characteristics of the sound wave transmitter, the upper and lower cutoff frequencies of the bandpass filter are set to ensure that only the echo signal that matches the frequency of the transmitted high-frequency sound wave is retained, and the digitized echo signal is bandpass filtered using the bandpass filter.

[0078] Specifically, in S32, the echo signal after bandpass filtering is preprocessed, including:

[0079] S321, aligning the bandpass filtered echo signal with the transmitted high-frequency sound wave, and normalizing them to the same amplitude range;

[0080] S322: Perform windowing processing on the normalized echo signal to reduce spectrum leakage.

[0081] Specifically, in S33, cross-correlation calculation is performed on the transmitted high-frequency sound wave and the pre-processed echo signal, and the cross-correlation calculation result is converted into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile, including:

[0082] S331. Directly perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal:

[0083]

[0084] Where R(τ) is the cross-correlation function value, which indicates the correlation between the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n] under the time shift parameter τ. τ is the time shift parameter, which indicates the time offset of the preprocessed echo signal y[n] relative to the transmitted high-frequency sound wave x[n]. n is a discrete time variable, which indicates the sampling point index of the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n]. N is the signal length, which indicates the total number of sampling points in the signal.

[0085] Or use FFT to accelerate the cross-correlation calculation between the transmitted high-frequency sound wave and the pre-processed echo signal:

[0086] R(τ)=IFFT(FFT(x[n])·conj(FFT(y[n])));

[0087] Where FFT(·) represents fast Fourier transform, conj(·) represents complex conjugate, and IFFT(·) represents inverse fast Fourier transform.

[0088] S332. Determine the time shift parameter corresponding to the maximum cross-correlation function value in the cross-correlation calculation result. If multiple maximum cross-correlation function values exist, determine a reasonable maximum cross-correlation function value based on prior knowledge.

[0089] S333. Calculate the distance between the acoustic wave receiver and the eyeball based on the time shift parameter corresponding to the maximum cross-correlation function value:

[0090]

[0091] Where d is the distance between the sound wave receiver and the eyeball, τ' is the time shift parameter corresponding to the maximum cross-correlation function value, and c is the speed of sound;

[0092] S334. According to the distance between the sound wave receiver and the eyeball and the position information of the sound wave receiver, a three-dimensional profile reflecting the position of the eyeball is determined by using triangulation or least squares method to generate an echo profile.

[0093] S4. Input the echo profile into the pre-trained eye tracking model. The eye tracking model extracts features from the echo profile and predicts the gaze point coordinates, such as Figure 3 As shown, specifically including:

[0094] S41, performing model training on the eye tracking model to obtain a pre-trained eye tracking model;

[0095] S42, preprocessing the echo profile, and inputting the preprocessed echo profile into a pre-trained eye tracking model;

[0096] S43. Utilize the residual block in the eye tracking model to extract deep features of the echo profile. The residual block alleviates the gradient vanishing problem through skip connections and improves feature extraction capabilities.

[0097] S44. Using the fully connected layer in the eye tracking model, the extracted deep features are mapped to the gaze point coordinates in the screen coordinate system or the three-dimensional space coordinate system;

[0098] Among them, the eye tracking model is built based on the residual network ResNet-18.

[0099] Specifically, in S41, the eye tracking model is trained to obtain a pre-trained eye tracking model, including:

[0100] S411, collecting echo profiles under different gaze directions, eye movement states, and head postures, as well as corresponding screen indicator point coordinates, annotating the echo profiles using the screen indicator point coordinates, and constructing a training data set;

[0101] S412, dividing the training data set into a training set, a validation set, and a test set according to a preset ratio;

[0102] S413, setting the loss function and optimizer of the eye tracking model;

[0103] S414, inputting the training set into the eye tracking model for model training;

[0104] S415, calculating the loss value based on the loss function, and the optimizer updating the model parameters according to the loss value and the network gradient information;

[0105] S416. If the loss value is less than the preset threshold, the model training ends and the current eye tracking model is the trained eye tracking model. Otherwise, the process returns to S414 and continues to train the model using the training set.

[0106] S417. Input the validation set into the trained eye tracking model, evaluate the generalization ability of the model by observing its performance on the validation set, and tune the model's hyperparameters and structure;

[0107] S418. Input the test set into the tuned eye tracking model to evaluate the model performance.

[0108] In the technical solution of this application, the signal processing module and the eye tracking model are integrated into the processing unit. The processing unit uses Teensy4.1 or MAX78002 as a microcontroller and integrates a low-power CNN accelerator to support 30-83.3Hz real-time eye tracking.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An eye tracking method based on high-frequency sound waves, characterized by: The following steps are involved: S1, the sound wave transmitter emits high-frequency sound waves to the eye area; S2, the sound wave receiver receives the echo signal formed by the reflection of the eyeball and surrounding tissues; S3, the signal processing module processes the echo signal and generates an echo profile according to the processed echo signal; S4. Input the echo profile into the pre-trained eye tracking model. The eye tracking model extracts features from the echo profile and predicts the gaze point coordinates.

2. The eye tracking method based on high-frequency sound waves according to claim 1, characterized in that: The signal processing module in S3 processes the echo signal and generates an echo profile based on the processed echo signal, including: S31, performing band-pass filtering on the echo signal; S32, preprocessing the echo signal after bandpass filtering; S33. Perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal, and convert the cross-correlation calculation result into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile.

3. The eye tracking method based on high-frequency sound waves according to claim 2, characterized in that: In S31, the echo signal is band-pass filtered, including: According to the frequency characteristics of the sound wave transmitter, the upper and lower cutoff frequencies of the bandpass filter are set to ensure that only the echo signal that matches the frequency of the transmitted high-frequency sound wave is retained, and the digitized echo signal is bandpass filtered using the bandpass filter.

4. The eye tracking method based on high-frequency sound waves according to claim 3, characterized in that: In S32, the echo signal after bandpass filtering is preprocessed, including: S321, aligning the bandpass filtered echo signal with the transmitted high-frequency sound wave, and normalizing them to the same amplitude range; S322: Perform windowing processing on the normalized echo signal to reduce spectrum leakage.

5. The eye tracking method based on high-frequency sound waves according to claim 4, characterized in that: In S33, a cross-correlation calculation is performed on the transmitted high-frequency sound wave and the pre-processed echo signal, and the cross-correlation calculation result is converted into a three-dimensional profile reflecting the position of the eyeball to generate an echo profile, including: S331. Directly perform cross-correlation calculation on the transmitted high-frequency sound wave and the pre-processed echo signal: Where R(τ) is the cross-correlation function value, which indicates the correlation between the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n] under the time shift parameter τ. τ is the time shift parameter, which indicates the time offset of the preprocessed echo signal y[n] relative to the transmitted high-frequency sound wave x[n]. n is a discrete time variable, which indicates the sampling point index of the transmitted high-frequency sound wave x[n] and the preprocessed echo signal y[n]. N is the signal length, which indicates the total number of sampling points in the signal. Or use FFT to accelerate the cross-correlation calculation between the transmitted high-frequency sound wave and the pre-processed echo signal: R(τ)=IFFT(FFT(x[n])·conj(FFT(y[n]))); Where FFT(·) represents fast Fourier transform, conj(·) represents complex conjugate, and IFFT(·) represents inverse fast Fourier transform. S332. Determine the time shift parameter corresponding to the maximum cross-correlation function value in the cross-correlation calculation result. If multiple maximum cross-correlation function values exist, determine a reasonable maximum cross-correlation function value based on prior knowledge. S333. Calculate the distance between the acoustic wave receiver and the eyeball based on the time shift parameter corresponding to the maximum cross-correlation function value: Where d is the distance between the sound wave receiver and the eyeball, τ' is the time shift parameter corresponding to the maximum cross-correlation function value, and c is the speed of sound; S334. According to the distance between the sound wave receiver and the eyeball and the position information of the sound wave receiver, a three-dimensional profile reflecting the position of the eyeball is determined by using triangulation or least squares method to generate an echo profile.

6. The eye tracking method based on high-frequency sound waves according to claim 1, wherein: In S4, the echo profile is input into the pre-trained eye tracking model. The eye tracking model extracts features from the echo profile and predicts the gaze point coordinates, including: S41, performing model training on the eye tracking model to obtain a pre-trained eye tracking model; S42, preprocessing the echo profile, and inputting the preprocessed echo profile into a pre-trained eye tracking model; S43. Utilize the residual block in the eye tracking model to extract deep features of the echo profile. The residual block alleviates the gradient vanishing problem through skip connections and improves feature extraction capabilities. S44. Using the fully connected layer in the eye tracking model, the extracted deep features are mapped to the gaze point coordinates in the screen coordinate system or the three-dimensional space coordinate system; Among them, the eye tracking model is built based on the residual network ResNet-18.

7. The eye tracking method based on high-frequency sound waves according to claim 6, characterized in that: In S41, the eye tracking model is trained to obtain a pre-trained eye tracking model, including: S411, collecting echo profiles under different gaze directions, eye movement states, and head postures, as well as corresponding screen indicator point coordinates, annotating the echo profiles using the screen indicator point coordinates, and constructing a training data set; S412, dividing the training data set into a training set, a validation set, and a test set according to a preset ratio; S413, setting the loss function and optimizer of the eye tracking model; S414, inputting the training set into the eye tracking model for model training; S415, calculating the loss value based on the loss function, and the optimizer updating the model parameters according to the loss value and the network gradient information; S416. If the loss value is less than the preset threshold, the model training ends and the current eye tracking model is the trained eye tracking model. Otherwise, the process returns to S414 and continues to train the model using the training set. S417. Input the validation set into the trained eye tracking model, evaluate the generalization ability of the model by observing its performance on the validation set, and tune the model's hyperparameters and structure; S418. Input the test set into the tuned eye tracking model to evaluate the model performance.

8. The eye tracking method based on high-frequency sound waves according to any one of claims 1 to 7, characterized in that: The sound wave transmitter includes a first speaker and a second speaker symmetrically embedded in two glasses legs. The first speaker transmits 18-21 kHz FMCW sound waves to the eye area, and the second speaker transmits 21.5-24.5 kHz FMCW sound waves to the eye area.

9. The eye tracking method based on high-frequency sound waves according to claim 8, characterized in that: The sound wave receiver includes 8 miniature microphones, and the 8 miniature microphones are symmetrically embedded in two glasses legs.

10. The eye tracking method based on high-frequency sound waves according to claim 9, characterized in that: The signal processing module and the eye tracking model are integrated into a processing unit. The processing unit adopts Teensy4.1 or MAX78002 as a microcontroller and integrates a low-power CNN accelerator.

Images