A flexible eye movement perception and tracking device and an eye movement signal processing method thereof
By combining a flexible nanofilm sensor and the autocorrelation function maximization method with the ResNet-50 neural network model, the problems of poor portability and insufficient real-time performance of existing eye-tracking devices are solved, achieving low-cost and high-accuracy eye-tracking perception and tracking, supporting real-time eye-controlled human-computer interaction and real-world object control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-14
- Publication Date
- 2026-06-02
AI Technical Summary
Existing eye-tracking devices are expensive, poorly portable, have weak anti-interference capabilities, and lack real-time performance, making it difficult to achieve low-cost, lightweight, real-time, and highly accurate eye-tracking perception and tracking.
A flexible nanofilm sensor was fabricated by encapsulating a patterned protein-silver nanocomposite film between polydimethylsiloxane molecules. This sensor was combined with the autocorrelation function maximization method and the ResNet-50 neural network model to achieve real-time detection and classification of eye movement signals.
It achieves non-invasive, portable, low-cost, and highly accurate eye movement perception and tracking, can decode eye movements in real time and support eye-controlled human-computer interaction, and is suitable for controlling hardware facilities such as virtual games, virtual keyboards and robotic arms.
Smart Images
Figure CN122131916A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of eye-tracking technology, specifically relating to a flexible eye-tracking sensing and tracking device and its eye-tracking signal processing method. Background Technology
[0002] Eye movement describes the dynamic changes in the pupil, intuitively reflecting a person's fixation point and eye movement trajectory. It is of great significance for describing the processes and characteristics of human visual perception, reproducing basic individual cognitive processing, studying information processing methods of the visual system, and diagnosing and treating multidisciplinary diseases. Eye-tracking technology monitors eye movements using specialized sensors to understand and interpret user behavior. This technology has been widely applied in various fields such as medical research, behavioral analysis, interface design, and monitoring driver attention.
[0003] Current eye-tracking devices mainly fall into two categories: screen-based and glasses-based. Screen-based devices integrate an eye tracker above or below a computer display, typically containing one or more infrared light sources and an infrared camera. Glasses-based devices miniaturize the relevant optical components and integrate them into the eyeglass frame. Both approaches are based on the pupil-corneal reflex principle, requiring dedicated image acquisition modules, resulting in high costs and poor portability, hindering large-scale application. Other research uses flexible sensors to collect electrooculogram (EOG) signals for eye movement detection, but this method suffers from weak signal strength, susceptibility to electromyographic interference, and requires eye movements to last no more than 2 seconds, resulting in insufficient real-time performance. Therefore, there is an urgent need to develop a low-cost, lightweight, portable, interference-resistant, and real-time-efficient eye-tracking system. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a flexible eye movement sensing and tracking device and its eye movement signal processing method, which realizes eye movement sensing and tracking with strong real-time performance, non-invasiveness, portability, low cost and high accuracy, and can be used for real-time decoding of eye movements and eye-controlled human-computer interaction.
[0005] The technical solution adopted to solve the above technical problems is: a flexible eye movement sensing and tracking device, including a flexible nanofilm sensor, which is attached to the upper eyelid and connected in series with a power supply and a voltage divider resistor to form a circuit. A sound card is connected in parallel with the flexible nanofilm sensor, and the sound card is electrically connected to a computer through a wire.
[0006] The flexible nanofilm sensor of the present invention is made by encapsulating a patterned protein-silver nanocomposite film between two layers of polydimethylsiloxane, and fixing copper wires to both ends of the flexible nanofilm sensor with conductive silver paste.
[0007] The patterned protein-silver nanocomposite film is formed by engraving a pattern on the protein-silver nanocomposite film using a laser engraving machine. The pattern is formed by alternating double helix lines with spokes arranged radially between the double helix lines. The width of the spokes between adjacent double helix lines is different, and the spokes are evenly arranged in a 360° phase. The shapes of adjacent spokes are inconsistent.
[0008] The method for preparing the protein-silver nanocomposite membrane of the present invention is as follows: equal volumes of 0.02-0.2 mg / mL protein aqueous solution, 5-60 mg / mL silver ammonia solution and 5-60 mg / mL glucose solution are mixed, coated on a glass substrate, and incubated in a humid environment at 5-40°C for 2-12 h to obtain the protein-silver nanocomposite membrane.
[0009] The protein of this invention is one of bovine serum albumin, α-amylase, collagen, keratin, pepsin, and ovalbumin.
[0010] The sound card of this invention has a sampling rate of over 190,000 times per second.
[0011] An eye movement signal processing method for a flexible eye movement sensing and tracking device includes the following steps:
[0012] Step 1: Connect the flexible nanofilm sensor, power supply, and voltage divider resistor in series to form a circuit. Connect the sound card and the flexible nanofilm sensor in parallel. Connect the sound card to the computer via wires. Apply the flexible nanofilm sensor to the upper eyelid to start collecting eye movement signals. Transmit the signals to the computer through the sound card.
[0013] Step 2: Divide the data signal acquired in Step 1 into frames, for a length of... The eye-tracking data is framed using the following formula:
[0014]
[0015] in, For frame length, For frame shifting, the data will be divided into frame;
[0016] Step 3: Perform median filtering on each frame of data from Step 2 using the following formula to remove noise;
[0017]
[0018] in, For the number of iterations, For the process The output signal after median filtering is used as the current input in each iteration, with the previous output serving as the input. For the first One data sample value;
[0019] Step 4: Calculate the maximum value of the autocorrelation function of each frame of data after filtering in Step 3 in real time, and set thresholds T1 and T2. When the maximum value of the autocorrelation function is always less than T1, it is determined that no valid eye movement signal has appeared. When the maximum value of the autocorrelation function jumps from less than or equal to T1 to greater than T2, it is determined that the eye movement signal has started. After determining that the start point has appeared, if the maximum value of the autocorrelation function of several consecutive frames is less than T1, it is determined that the eye movement signal has ended.
[0020] Step 5: Based on the start and end indexes obtained in Step 4, extract the data between the start and end points of the eye movement signal to obtain the single-cycle eye movement time series T= ;
[0021] Step 6: Extract the single-cycle eye movement data extracted in Step 5 using a relative position matrix, and convert it into a two-dimensional image format. The relative position matrix contains redundant information from the original time series, meaning that each row and column uses a certain timestamp as a reference point and contains information from the entire eye movement time series.
[0022] Step 7: Input the two-dimensional image format generated in Step 6 into the pre-trained ResNet-50 neural network model for classification. Different eye-tracking data are assigned a unique label corresponding to the action category after classification.
[0023] In steps 8 and 7, different category labels are encoded into different instructions. The program automatically determines whether the label matches the preset action instruction of the controlled object. If the match is successful, the action control program corresponding to the label is called to drive the controlled object to perform the target action, thereby realizing eye-controlled human-computer interaction.
[0024] Frame length of the present invention The frame shift is 8–16 ms. It takes 4 to 8 ms.
[0025] The autocorrelation function of this invention for:
[0026]
[0027] in, The delay value is 15 to 31. It is data for each frame.
[0028] The single-cycle eye-tracking data feature extraction of the present invention includes the following steps:
[0029] Step 6.1: Perform z-score normalization on the single-cycle eye-tracking data to obtain the standard normal distribution Z. l :
[0030]
[0031] in, yes The average value, yes Standard deviation, t l It is the value at timestamp l, where l takes the form of a finite positive integer;
[0032] Step 6.2: Select the dimensionality reduction factor r, and use the piecewise aggregation approximation method to reduce Z... l The dimension is reduced to p, generating a new eye-tracking time series. By calculating the average of a piecewise constant, the normalized eye-tracking time series is transformed from... The dimension is reduced to p-dimensional while maintaining the approximate trend of the original eye-tracking sequence, where r ranges from 2 to 4;
[0033] Step 6.3: Construct a p×p matrix, calculate the relative position between two timestamps, and combine the eye-tracking time series. Convert to a two-dimensional matrix P
[0034]
[0035] In this process, every two timestamps in the eye-tracking time series are connected by P to obtain their relative positions. Each row and column of P takes a certain timestamp as a reference point and contains information about the entire eye-tracking time series.
[0036] Step 6.4: Convert the two-dimensional matrix P into a grayscale matrix using the following formula, and assign a different color to each element in the grayscale matrix, ultimately displaying it as an image.
[0037] .
[0038] The pre-trained ResNet-50 neural network model classification of this invention includes the following steps:
[0039] Step 7.1: The input image first passes through an initial convolutional layer with a 7×7 kernel for basic feature extraction. After normalization, non-linear features are introduced through the ReLU activation function.
[0040] Step 7.2: Use a 3×3 max pooling layer to downsample the feature map, reducing the feature map size to 56×56;
[0041] Step 7.3: Input the feature map into the residual network consisting of 4 stages. Each stage has 3, 4, 6, and 3 Bottleneck structure residual blocks, which are stacked and skipped through 1×1, 3×3, and 1×1 convolutional layers.
[0042] Step 7.4: At the end of the network, a global average pooling layer is used to compress the feature map into a 1×1×2048 channel vector;
[0043] Step 7.5: Output the classification probability through a custom fully connected layer, compare the difference between the model output and the true label, and quantify the error by calculating the cross-entropy loss function;
[0044] Step 7.6: Starting from the loss function, calculate the gradient of the parameters of each layer using the chain rule, and use the momentum gradient descent optimizer to update the model parameters based on the gradient.
[0045] Compared with the prior art, the present invention has the following advantages:
[0046] 1. The flexible nanofilm sensor used in this invention has a thickness of 1.6 to 2.4 mm, a weight of 1 to 2 g, and a width and length of 24 mm. After being patterned by laser engraving, it has a compact structure, high sensitivity, low cost, non-invasiveness, lightness, softness, and stability, effectively overcoming the problems of poor portability and high price of traditional eye trackers.
[0047] 2. This invention only requires the collection of sensor data from a single channel to complete eye tracking, eliminating the need to attach sensors to multiple locations around the eye.
[0048] 3. The autocorrelation function maximum method used in this invention can detect eye movement signals in real time and automatically complete the segmentation and extraction of individual eye movement data, overcoming the shortcomings of traditional methods such as poor real-time performance or the need for manual segmentation of eye movement data.
[0049] 4. This invention employs the relative position matrix method, which transforms one-dimensional eye-tracking data into two-dimensional images more suitable for convolutional neural network processing while completing feature extraction. This significantly improves classification accuracy and is simple to operate and highly real-time, providing a reference for data processing methods in other fields.
[0050] 5. This invention uses the ResNet-50 neural network model for eye movement classification, overcoming the shortcomings of low classification accuracy in traditional methods, and enabling real-time and accurate feedback of eye movement classification results.
[0051] 6. This invention realizes real-time eye-controlled human-computer interaction. Experimental verification shows that the flexible eye-tracking and sensing device can not only realize eye-tracking control of software such as virtual games and virtual keyboards, but also be used in combination with hardware facilities such as robotic arms, so that eye-tracking control is no longer limited to virtual scenes, but can also realize the control of real-world objects. Attached Figure Description
[0052] Figure 1 This is a schematic diagram of the hardware connection of the flexible eye-tracking sensing and tracking device of the present invention;
[0053] Figure 2 This is a schematic diagram illustrating the use of sensors to acquire signals according to the present invention;
[0054] Figure 3 This is a schematic diagram of the sensor structure and manufacturing process used in this invention;
[0055] Figure 4 Here are schematic diagrams and physical images of the double-helix spoke structure of the sensor used in this invention;
[0056] Figure 5 This is a flowchart of the eye movement data processing of the flexible eye movement sensing and tracking device of the present invention;
[0057] Figure 6 The results of real-time signal segmentation using the autocorrelation function maximum method in this invention are shown. The solid line represents the starting point of the eye movement signal, and the dashed line represents the ending point of the eye movement signal.
[0058] Figure 7 a, 7b, 7c, and 7d are two-dimensional images of eye movement data (up, down, left, and right) collected by the sensor used in this invention.
[0059] Figure 8 This is a confusion matrix of eye movements in eight directions (up, down, left, right, upper left, lower left, upper right, and lower right) collected by the sensor used in this invention.
[0060] Figure 9 This is a diagram of the ResNet-50 neural network structure used in this invention;
[0061] Figure 10 a, 10b, and 10c represent the operating system framework for flexible eye-tracking perception and tracking devices to control virtual games, virtual keyboards, and robotic arms. Detailed Implementation
[0062] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to these embodiments.
[0063] Example 1
[0064] exist Figure 1 , 2The present invention relates to a flexible eye movement sensing and tracking device, comprising a flexible nanofilm sensor, which is attached to the upper eyelid. The flexible nanofilm sensor, a power supply, and a voltage divider resistor are connected in series to form a circuit. A sound card is connected in parallel with the flexible nanofilm sensor. The sound card is electrically connected to a computer via a wire. The sound card collects the voltage of the voltage divider resistor at a sampling frequency of 1000 Hz and transmits the collected voltage signal to the computer for processing. In this embodiment, the sampling rate of the sound card is higher than 190,000 times / second, the resistance value is 1 Ω, and the power supply voltage is 1 to 3 V.
[0065] like Figure 3 As shown, the flexible nanofilm sensor is fabricated by encapsulating a patterned protein-silver nanocomposite film between two layers of polydimethylsiloxane, and fixing copper wires to both ends of the sensor with conductive silver paste. The patterned protein-silver nanocomposite film is created by engraving patterns onto the protein-silver nanocomposite film using a laser engraving machine, such as... Figure 4 As shown, the pattern is formed by alternating double helix lines, with spokes radially arranged between the double helix lines. The spoke widths differ between adjacent spacings of the double helix lines, and the spokes are uniformly arranged in a 360° phase, with adjacent spokes having different shapes. The preparation and encapsulation of the protein-silver nanocomposite film were carried out using the method disclosed in the literature "Rongrong Qin, Yongchun Liu, Fei Tao, Chen Li, Wenfei Cao, Peng Yang*. Protein-bound free-standing 2D metal film for stealth information transmission. Adv. Mater. 2019, 31, 1803377.", wherein the flexible nanofilm has a length and width of 24×24 mm, the PDMS has a length and width of 25×25 mm, and the total thickness of the encapsulated throat vibration sensor is 1.6–2.4 mm.
[0066] The protein-silver nanocomposite membrane prepared in this embodiment is prepared by mixing equal volumes of 0.2 mg / mL bovine serum albumin aqueous solution, 30 mg / mL silver ammonia solution and 30 mg / mL glucose solution, coating the mixture onto a glass substrate, and incubating it in a humid environment at room temperature for 8 hours to obtain the protein-silver nanocomposite membrane.
[0067] like Figure 5 As shown, the present invention discloses an eye movement signal processing method for a flexible eye movement sensing and tracking device, comprising the following steps:
[0068] Step 1: Connect the flexible nanofilm sensor, power supply, and voltage divider resistor in series to form a circuit. Connect the sound card and the flexible nanofilm sensor in parallel. Connect the sound card to the computer via wires. Apply the flexible nanofilm sensor to the upper eyelid to start collecting eye movement signals. Transmit the signals to the computer through the sound card. In this embodiment, the flexible nanofilm sensor has a thickness of 1.8 mm, a weight of 1.2 g, and a width and length of 24 mm.
[0069] Step 2: Divide the data signal acquired in Step 1 into frames, for a length of... The eye-tracking data is framed using the following formula:
[0070]
[0071] in, For frame length, For frame shifting, the data will be divided into Frame; Frame length in this embodiment 10ms, frame shift The time is 5 ms, and the window function uses a Hamming window.
[0072] Step 3: Perform median filtering on each frame of data from Step 2 using the following formula to remove noise.
[0073]
[0074] in, For the number of iterations, For the process The output signal after median filtering is used as the current input in each iteration, with the previous output serving as the input. For the first Each data sample value; in this embodiment, median filtering with a window size of 10 ms and 10 iterations is performed on each frame of data to remove noise.
[0075] Step 4: Calculate the maximum value of the autocorrelation function of each frame of data after filtering in Step 3 in real time. for:
[0076]
[0077] in, The delay value is 15 to 31. It is data for each frame.
[0078] Thresholds T1 and T2 are set. When the maximum value of the autocorrelation function is always less than T1, it is determined that no valid eye movement signal has appeared. When the maximum value of the autocorrelation function jumps from less than or equal to T1 to greater than T2, it is determined that the eye movement signal has started. After determining that the start point has appeared, if the maximum value of the autocorrelation function is less than T1 for several consecutive frames, it is determined that the eye movement signal has ended. In this embodiment, thresholds are set. , The segmentation results are as follows Figure 6 As shown.
[0079] Step 5: Based on the start and end indexes obtained in Step 4, extract the data between the start and end points of the eye movement signal to obtain the single-cycle eye movement time series T= ;
[0080] Step 6: Extract features from the single-cycle eye-tracking data extracted in Step 5 using a relative position matrix and convert it into a two-dimensional image format. The relative position matrix contains redundant information from the original time series, meaning that each row and column uses a specific timestamp as a reference point and contains information from the entire eye-tracking time series, which can more clearly present intra-class similarity and inter-class difference features.
[0081] The single-cycle eye-tracking data feature extraction includes the following steps:
[0082] Step 6.1: Perform z-score normalization on the single-cycle eye-tracking data to obtain the standard normal distribution Z. l :
[0083]
[0084] in, yes The average value, yes Standard deviation, t l It is the value at timestamp l, where l takes the form of a finite positive integer;
[0085] Step 6.2: Select the dimensionality reduction factor r, and use the piecewise aggregation approximation method to reduce Z... l The dimension is reduced to p, generating a new eye-tracking time series. By calculating the average of a piecewise constant, the normalized eye-tracking time series is transformed from... The dimension is reduced to p-dimensional while maintaining the approximate trend of the original eye-tracking sequence, where r ranges from 2 to 4;
[0086] Step 6.3: Construct a p×p matrix, calculate the relative position between two timestamps, and combine the eye-tracking time series. Convert to a two-dimensional matrix P
[0087]
[0088] In this process, every two timestamps in the eye-tracking time series are connected by P to obtain their relative positions. Each row and column of P takes a certain timestamp as a reference point and contains information about the entire eye-tracking time series.
[0089] Step 6.4: Convert the two-dimensional matrix P into a grayscale matrix using the following formula, and assign a different color to each element in the grayscale matrix, ultimately displaying it as an image, such as... Figure 7 As shown.
[0090] .
[0091] Step 7: Input the two-dimensional image format generated in Step 6 into the pre-trained ResNet-50 neural network model for classification. Different eye-tracking data are assigned a unique label corresponding to the action category after classification.
[0092] like Figure 9 As shown, the pre-trained ResNet-50 neural network model classification described above includes the following steps:
[0093] Step 7.1: The input image first passes through an initial convolutional layer with a 7×7 kernel for basic feature extraction. After normalization, non-linear features are introduced through the ReLU activation function.
[0094] Step 7.2: Use a 3×3 max pooling layer to downsample the feature map, reducing the feature map size to 56×56;
[0095] Step 7.3: Input the feature map into the residual network consisting of 4 stages. Each stage has 3, 4, 6, and 3 Bottleneck structure residual blocks, which are stacked and skipped through 1×1, 3×3, and 1×1 convolutional layers.
[0096] Step 7.4: At the end of the network, a global average pooling layer is used to compress the feature map into a 1×1×2048 channel vector;
[0097] Step 7.5: Output the classification probability through a custom fully connected layer, compare the difference between the model output and the true label, and quantify the error by calculating the cross-entropy loss function;
[0098] Step 7.6: Starting from the loss function, calculate the gradient of the parameters of each layer using the chain rule, and use the momentum gradient descent optimizer to update the model parameters based on the gradient.
[0099] In this embodiment, the generated images (Pushing Box dataset) are input into the ResNet-50 neural network model to obtain the Pushing Box pre-trained model. The Pushing Box pre-trained model is used for classification. The eye movement data is divided into four categories: up, down, left, and right, corresponding to the player moving up, down, left, and right. After classification, different eye movement data are assigned a unique label corresponding to the action category.
[0100] In steps 8 and 7, different category labels are encoded into different instructions. The program automatically determines whether the label matches the preset action instruction of the controlled object. If the match is successful, the action control program corresponding to the label is called to drive the controlled object to perform the target action, thereby realizing eye-controlled human-computer interaction.
[0101] In this implementation, the user controls the player's movement through eye tracking. Each eye movement generates a two-dimensional image, which is fed into a pre-trained Sokoban neural network model for classification, obtaining category labels. These labels are compared with control programs that move the box in different directions. If they match, the corresponding control program is invoked, causing the player to move the box in that direction. The game ends when the box coincides with the finish line. Figure 10 As shown in a.
[0102] Example 2
[0103] In the above embodiment 1, the flexible nanofilm sensor in step 1 of this embodiment has a thickness of 2.0 mm, a weight of 1.5 g, and a width and length of 24 mm.
[0104] In step 7, the generated images (robotic arm dataset) are input into the ResNet-50 neural network model to obtain the robotic arm pre-trained model. The robotic arm pre-trained model is used for classification. The eye movement data types are up, down, left, right, three consecutive blinks, and two consecutive blinks, which correspond to the robotic arm raising, lowering, turning left, turning right, and the opening and closing of the gripper. After classification, different eye movement data are assigned a unique label corresponding to the action category.
[0105] In step eight, the user controls the robotic arm's movement via eye tracking. Each eye movement generates a two-dimensional image, which is fed into a pre-trained neural network model for classification, resulting in category labels. These labels are then compared with the control programs that control the robotic arm's movement in different directions and the opening and closing of the gripper. If they match, the corresponding control program is invoked, causing the robotic arm to perform the appropriate action. Figure 10 As shown in c.
[0106] Example 3
[0107] In the above embodiment 1, the flexible nanofilm sensor in step 1 of this embodiment has a thickness of 2.2 mm, a weight of 1.8 g, and a width and length of 24 mm.
[0108] In step seven, the generated images (keyboard dataset) are input into the ResNet-50 neural network model to obtain the keyboard pre-trained model. This pre-trained model is then used for classification, and the classification results are as follows: Figure 8 As shown, the eye movement data types are up, down, left, right, and two consecutive blinks. Up, down, left, and right correspond to the cursor moving up, down, left, and right, respectively; two consecutive blinks correspond to confirmation of selection. Different eye movement data are assigned a unique label corresponding to the action category after classification.
[0109] In step eight, the user controls the cursor movement via eye tracking. Each eye movement generates a two-dimensional image, which is fed into a pre-trained virtual keyboard model for classification to obtain category labels. These labels are compared with the program controlling the cursor; if they match, the corresponding control program is invoked to move the cursor in the corresponding direction or confirm selection. Figure 10 As shown in b.
Claims
1. A flexible eye-tracking sensing and tracking device, characterized in that: The device includes a flexible nanofilm sensor, which is attached to the upper eyelid and connected in series with a power supply and a voltage divider resistor to form a circuit. A sound card is connected in parallel with the flexible nanofilm sensor, and the sound card is electrically connected to a computer via wires.
2. The flexible eye-tracking sensing and tracking device according to claim 1, characterized in that: The flexible nanofilm sensor is made by encapsulating a patterned protein-silver nanocomposite film between two layers of polydimethylsiloxane, and fixing copper wires to both ends of the flexible nanofilm sensor with conductive silver paste. The patterned protein-silver nanocomposite film is formed by engraving a pattern on the protein-silver nanocomposite film using a laser engraving machine. The pattern is formed by alternating double helix lines with spokes arranged radially between the double helix lines. The width of the spokes between adjacent double helix lines is different, and the spokes are evenly arranged in a 360° phase. The shapes of adjacent spokes are inconsistent.
3. The flexible eye-tracking sensing and tracking device according to claim 2, characterized in that, The method for preparing the protein-silver nanocomposite film is as follows: equal volumes of 0.02-0.2 mg / mL protein aqueous solution, 5-60 mg / mL silver ammonia solution and 5-60 mg / mL glucose solution are mixed, coated on a glass substrate, and incubated in a humid environment at 5-40℃ for 2-12 hours to obtain the protein-silver nanocomposite film.
4. The flexible eye-tracking sensing and tracking device according to claim 3, characterized in that: The protein in question is one of bovine serum albumin, α-amylase, collagen, keratin, pepsin, or ovalbumin.
5. The flexible eye-tracking sensing and tracking device according to claim 1, characterized in that: The sound card has a sampling rate of over 190,000 times per second.
6. A method for processing eye movement signals in a flexible eye movement sensing and tracking device, characterized in that... Includes the following steps: Step 1: Connect the flexible nanofilm sensor, power supply, and voltage divider resistor in series to form a circuit. Connect the sound card and the flexible nanofilm sensor in parallel. Connect the sound card to the computer via wires. Apply the flexible nanofilm sensor to the upper eyelid to start collecting eye movement signals. Transmit the signals to the computer through the sound card. Step 2: Divide the data signal acquired in Step 1 into frames, for a length of... The eye-tracking data is framed using the following formula: in, For frame length, For frame shifting, the data will be divided into frame; Step 3: Perform median filtering on each frame of data from Step 2 using the following formula to remove noise; in, For the number of iterations, For the process The output signal after median filtering is used as the current input in each iteration, with the previous output serving as the input. For the first One data sample value; Step 4: Calculate the maximum value of the autocorrelation function of each frame of data after filtering in Step 3 in real time, and set thresholds T1 and T2. When the maximum value of the autocorrelation function is always less than T1, it is determined that no valid eye movement signal has appeared. When the maximum value of the autocorrelation function jumps from less than or equal to T1 to greater than T2, it is determined that the eye movement signal has started. After determining that the start point has appeared, if the maximum value of the autocorrelation function of several consecutive frames is less than T1, it is determined that the eye movement signal has ended. Step 5: Based on the start and end indexes obtained in Step 4, extract the data between the start and end points of the eye movement signal to obtain the single-cycle eye movement time series T= ; Step 6: Extract the single-cycle eye movement data extracted in Step 5 using a relative position matrix, and convert it into a two-dimensional image format. The relative position matrix contains redundant information from the original time series, meaning that each row and column uses a certain timestamp as a reference point and contains information from the entire eye movement time series. Step 7: Input the two-dimensional image format generated in Step 6 into the pre-trained ResNet-50 neural network model for classification. Different eye-tracking data are assigned a unique label corresponding to the action category after classification. In steps 8 and 7, different category labels are encoded into different instructions. The program automatically determines whether the label matches the preset action instruction of the controlled object. If the match is successful, the action control program corresponding to the label is called to drive the controlled object to perform the target action, thereby realizing eye-controlled human-computer interaction.
7. The eye movement signal processing method for a flexible eye movement sensing and tracking device according to claim 6, characterized in that: The frame length The frame shift is 8–16 ms. It takes 4 to 8 ms.
8. The eye movement signal processing method of a flexible eye movement sensing and tracking device according to claim 6, characterized in that... The autocorrelation function for: in, The delay value is 15 to 31. It is data for each frame.
9. The eye movement signal processing method of a flexible eye movement sensing and tracking device according to claim 6, characterized in that... The single-cycle eye-tracking data feature extraction includes the following steps: Step 6.1: Perform z-score normalization on the single-cycle eye-tracking data to obtain the standard normal distribution Z. l : in, yes The average value, yes Standard deviation, t l It is the value at timestamp l, where l takes the form of a finite positive integer; Step 6.2: Select the dimensionality reduction factor r, and use the piecewise aggregation approximation method to reduce Z... l The dimension is reduced to p, generating a new eye-tracking time series. By calculating the average of a piecewise constant, the normalized eye-tracking time series is transformed from... The dimension is reduced to p-dimensional while maintaining the approximate trend of the original eye-tracking sequence, where r ranges from 2 to 4; Step 6.3: Construct a p×p matrix, calculate the relative position between two timestamps, and combine the eye-tracking time series. Convert to a two-dimensional matrix P In this process, every two timestamps in the eye-tracking time series are connected by P to obtain their relative positions. Each row and column of P takes a certain timestamp as a reference point and contains information about the entire eye-tracking time series. Step 6.4: Convert the two-dimensional matrix P into a grayscale matrix using the following formula, and assign a different color to each element in the grayscale matrix, ultimately displaying it as an image. 。 10. The eye movement signal processing method of a flexible eye movement sensing and tracking device according to claim 6, characterized in that... The pre-trained ResNet-50 neural network model classification includes the following steps: Step 7.1: The input image first passes through an initial convolutional layer with a 7×7 kernel for basic feature extraction. After normalization, non-linear features are introduced through the ReLU activation function. Step 7.2: Use a 3×3 max pooling layer to downsample the feature map, reducing the feature map size to 56×56; Step 7.3: Input the feature map into the residual network consisting of 4 stages. Each stage has 3, 4, 6, and 3 Bottleneck structure residual blocks, which are stacked and skipped through 1×1, 3×3, and 1×1 convolutional layers. Step 7.4: At the end of the network, a global average pooling layer is used to compress the feature map into a 1×1×2048 channel vector; Step 7.5: Output the classification probability through a custom fully connected layer, compare the difference between the model output and the true label, and quantify the error by calculating the cross-entropy loss function; Step 7.6: Starting from the loss function, calculate the gradient of the parameters of each layer using the chain rule, and use the momentum gradient descent optimizer to update the model parameters based on the gradient.