An intelligent interactive control method and system for electronic photo frame based on eye tracking

Abstract

The present invention relates to the field of electronic photo frames and discloses a method and system for intelligent interactive control of electronic photo frames based on eye tracking. The method comprises: real-time tracking of the user's eyeball position based on eye image data, obtaining an eye movement trajectory sequence based on changes in pupil center coordinates and eyelid opening and closing; calculating gaze point coordinates, gaze duration, and saccade frequency based on the eye movement trajectory sequence; obtaining the user's current eye movement behavior type through a personalized recognition model; matching the user's current eye movement behavior type with a preset eye movement pattern library to obtain a set of candidate commands; and executing photo frame functions according to the control commands. This method has the following advantages: contactless intelligent control based on eye movements improves the convenience and accuracy of human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of electronic photo frames, and in particular to an intelligent interactive control method and system for electronic photo frames based on eye tracking. Background Art

[0002] With the rapid development of human-computer interaction technology, the interaction methods of smart display devices are undergoing profound changes. Eye tracking, as an emerging contactless interaction technology, has shown great application potential and market value in consumer electronics such as electronic photo frames.

[0003] Most electronic photo frames currently on the market use traditional interaction methods, such as touchscreens, physical buttons, or remote controls. These methods have significant shortcomings in specific scenarios. For example, touchscreen operations easily leave fingerprints on the screen, affecting the viewing experience. Physical buttons not only increase device complexity but also the likelihood of malfunction. Remote controls are easily lost and require additional storage space.

[0004] Applying eye tracking technology to the intelligent interactive control of electronic photo frames presents numerous technical challenges in designing and recognizing eye movement commands. Users' eye movements vary significantly from person to person and can change with varying usage scenarios. The same eye movement can convey completely different operational intentions in different situations. This uncertainty in intent recognition directly impacts the system's ability to accurately assess the user's true operational intent.

[0005] How to solve the above technical problems is a technical difficulty that needs to be overcome by those skilled in the art. Summary of the Invention

[0006] The present invention provides an eye-tracking-based intelligent interactive control method and system for an electronic photo frame, which at least partially solves the above technical problems.

[0007] In a first aspect, in order to solve the above technical problems, the present invention provides an intelligent interactive control method for an electronic photo frame based on eye tracking, comprising:

[0008] The eyeball position is tracked in real time based on the user's eye image data, and the eye movement trajectory sequence is obtained according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state;

[0009] Calculating the gaze point coordinates, gaze duration, and eye saccade frequency based on the eye movement trajectory sequence; obtaining the user's current eye movement behavior type through a personalized recognition model; and matching the user's current eye movement behavior type with a preset eye movement pattern library to obtain a candidate instruction set.

[0010] If there are multiple candidate instructions in the candidate instruction set, the user's historical operation preferences and current interface state information are analyzed to prioritize the candidate instructions; the candidate instruction with the highest priority is determined as the user's actual operation intention and a corresponding control instruction is generated;

[0011] Execute a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo-type eye movement behavior is identified after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0012] In an optional embodiment, the step of constructing the personalized recognition model includes:

[0013] Obtaining error samples and correct operation samples of the user to form a data set for building a personalized eye movement model; the data set includes a number of eye movement features, the eye movement features including gaze point coordinates, gaze duration, and eye saccade frequency;

[0014] Extracting the eye movement features from the error samples and the correct operation samples and constructing an initial eye movement model;

[0015] Analyzing the eye movement features, calculating the weight coefficient of each eye movement feature to obtain a feature weight distribution;

[0016] Determining whether the feature weight distribution deviates from a preset threshold, and if so, adjusting the weight coefficient according to the user's operating habits to generate an optimized eye movement model;

[0017] Based on the optimized eye movement model, the command recognition threshold is adjusted in combination with the user's operating habits to obtain a personalized recognition model;

[0018] Processing the real-time eye movement data through the personalized recognition model to determine whether the command recognition result conforms to the user's operating habits and generate a matching result; updating the eye movement feature weight and command recognition threshold of the personalized recognition model according to the matching result;

[0019] If the matching result of the personalized recognition model does not reach the preset accuracy, the previous step is repeated to optimize the model parameters until the final personalized recognition model is obtained.

[0020] In an optional embodiment, the method further includes:

[0021] Collect user interaction data in real time through the system background; extract user identification information and historical recognition accuracy based on the user interaction data; trigger progressive learning mode if the user's recognition accuracy is lower than the preset threshold or the user is a newly registered user;

[0022] In the progressive learning mode, core functional instructions are selected from a preset instruction library to generate a simplified instruction sequence based on the user's current operating level. A corresponding visual guidance element is generated for each simplified instruction. The form and intensity of visual feedback are adjusted based on the user's eye movement characteristics to match the user's operating habits. The simplified instruction sequence covers the most basic and commonly used interactive operations.

[0023] When users use simplified instructions, eye movement and operation data are collected in real time to calculate the user's proficiency index. When the user's proficiency reaches the proficiency threshold, it is determined that the user has mastered the current instruction set.

[0024] After determining that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of new instructions.

[0025] In an optional embodiment, the method further includes:

[0026] Analyze scenarios where user command execution fails to identify the error type and its cause;

[0027] Adjust the parameters of the personalized recognition model based on the distribution of error types and adjust the instruction mapping rules based on the user's operating habits.

[0028] In an optional embodiment, the eyeball position is tracked in real time based on the user's eye image data, and an eye movement trajectory sequence is obtained according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state, including:

[0029] Collect user eye image data through the camera module;

[0030] Performing image preprocessing on the acquired image to obtain a first image;

[0031] Detecting the pupil area of ​​the first image based on the YOLOv5 model of deep learning to obtain the pupil bounding box coordinates and calculating the pupil center coordinates according to the bounding box center;

[0032] Detect the distance between the upper and lower eyelids to determine the eyelid opening and closing status;

[0033] Generate a time-stamped eye movement trajectory sequence based on the displacement changes of pupil center coordinates and the time series of eyelid opening and closing states in consecutive frames; wherein, when the eye movement trajectory sequence is interrupted, a re-localization mechanism is triggered to rescan the first image until stable pupil coordinates are obtained;

[0034] The trajectory sequence is smoothed using the Kalman filter algorithm to obtain a second trajectory sequence;

[0035] The Euclidean distance of pupil displacement between adjacent frames of the second trajectory sequence is detected to identify abnormal points, and the trajectory breakpoints are repaired using interpolation to form a continuous and stable third trajectory sequence;

[0036] Eye movement feature parameters are extracted based on the pupil center coordinate change rate and eyelid opening and closing frequency of the third trajectory sequence.

[0037] In an optional embodiment, the gaze point coordinates, gaze duration, and eye saccade frequency are calculated based on the eye movement trajectory sequence; the user's current eye movement behavior type is obtained through a personalized recognition model; and the user's current eye movement behavior type is matched with a preset eye movement pattern library to obtain a candidate instruction set, including:

[0038] Calculate the fixation dwell time, saccade speed, and blink frequency based on the eye movement trajectory sequence:

[0039] Based on the personalized recognition model, the calculated feature vector is compared with the preset eye movement pattern library, and the K-nearest neighbor algorithm is used to calculate the similarity between the current feature vector and each pattern in the pattern library;

[0040] If the similarity between the features within a certain time period and a certain pattern in the pattern library exceeds a preset threshold, it is determined to be the corresponding eye movement behavior type;

[0041] For the identified eye movement behavior type, a set of candidate instruction sets associated with it is filtered out from the preset instruction library through mapping relationships.

[0042] In an optional embodiment, if there are multiple candidate instructions in the candidate instruction set, the user's historical operation preferences and current interface state information are analyzed to prioritize the candidate instructions; the candidate instruction with the highest priority is determined as the user's actual operation intention and a corresponding control instruction is generated, including:

[0043] Obtain user historical operation data and current interface status information;

[0044] Standardize the collected operation data and interface status information to form an input data set;

[0045] Using the Bayesian probability model to calculate the initial probability distribution of each candidate instruction based on the standardized input data set to obtain the probability value of each instruction;

[0046] Prioritizing the candidate instructions using a weighted summation method according to preset historical preference weights and interface state relevance weights to generate a sorted instruction priority list;

[0047] If the probability value of the highest priority instruction in the instruction priority list exceeds the preset probability value, it is regarded as the user's actual operation intention;

[0048] Generate corresponding control instructions based on the determined user's actual operation intention.

[0049] In an optional embodiment, a photo frame function operation is executed according to the control instruction; and a subsequent eye movement behavior of the user is monitored to determine whether the result of the executed photo frame function operation meets expectations. If an undo eye movement behavior is identified after the photo frame function operation is executed, the executed photo frame function operation is canceled and the corresponding control instruction is marked as an error sample, including:

[0050] Get control instructions and call the photo frame function module to perform corresponding operations and update the interface status; synchronously record operation logs, which include user ID, instruction content, execution status and timestamp information;

[0051] After executing the control command, the eye tracking device is started to collect the user's gaze point coordinate sequence in real time; the eye movement data within a preset time period after the operation is extracted for analysis;

[0052] Cluster analysis is performed on the collected eye movement data, and the K-means algorithm is used to divide the gaze area. The Euclidean distance between the current gaze center and the preset undo area in the interface is calculated. If the gaze center is within the undo area and the gaze duration exceeds the preset duration, it is determined to be an undo-type eye movement behavior.

[0053] If an undo eye movement behavior is identified, it is determined that the execution of this control instruction does not meet the user's expectations; the executed photo frame function operation is undoed according to the instruction identifier in the operation log;

[0054] The control instruction and its associated eye movement behavior data are marked as error samples and stored in the error sample database.

[0055] In an optional embodiment, the method further includes:

[0056] Obtaining the user's most recent several operation sequences; using the most recent several operation sequences as context input;

[0057] Initialize the long short-term memory neural network model;

[0058] The most recent several operation sequences are input into the LSTM model; the LSTM model learns and extracts the temporal dependencies between operations and outputs a vector representing the contextual features of the current operation sequence; the vector represents the historical pattern of user operations and their temporal correlation.

[0059] Get the status information of the current interface in real time;

[0060] The operation context feature vector output by the LSTM model is concatenated with the current interface state information to form a new comprehensive feature vector.

[0061] The new comprehensive feature vector is input into the Bayesian probability model to calculate the probability of the candidate instruction.

[0062] In a second aspect, the present invention provides an electronic photo frame intelligent interactive control system based on eye tracking, comprising:

[0063] The first processing module is used to: track the eyeball position in real time based on the user's eye image data, and obtain an eye movement trajectory sequence according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state;

[0064] The second processing module is configured to: calculate the gaze point coordinates, gaze duration, and eye saccade frequency based on the eye movement trajectory sequence; obtain the user's current eye movement behavior type through a personalized recognition model; and match the user's current eye movement behavior type with a preset eye movement pattern library to obtain a candidate instruction set;

[0065] A third processing module is configured to: if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface state information to prioritize the candidate instructions; determine the candidate instruction with the highest priority as the user's actual operation intention and generate a corresponding control instruction;

[0066] The fourth processing module is used to: execute a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo-type eye movement behavior is identified after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0067] Compared with the prior art, the present invention has at least the following beneficial effects: contactless intelligent control based on eye movements improves the convenience and accuracy of human-computer interaction of the electronic photo frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 1 is a flow chart of an intelligent interactive control method for an electronic photo frame based on eye tracking provided by a first embodiment of the present invention;

[0069] Figure 2 This is a block diagram of an electronic photo frame intelligent interactive control system based on eye tracking provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0071] Reference Figure 1 The first embodiment of the present invention provides an intelligent interactive control method for an electronic photo frame based on eye tracking, comprising the following steps:

[0072] S101, tracking the eyeball position in real time based on the user's eye image data, and obtaining an eye movement trajectory sequence according to changes in pupil center coordinates and eyelid opening and closing states;

[0073] S102, calculating the gaze point coordinates, gaze duration, and eye saccade frequency based on the eye movement trajectory sequence; obtaining the user's current eye movement behavior type through a personalized recognition model; matching the user's current eye movement behavior type with a preset eye movement pattern library to obtain a candidate instruction set;

[0074] S103, if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface state information to prioritize the candidate instructions; determine the candidate instruction with the highest priority as the user's actual operation intention and generate a corresponding control instruction;

[0075] S104: Execute a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo eye movement behavior is identified after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0076] In one embodiment, the step of constructing the personalized recognition model includes:

[0077] The user's error samples and correct operation samples are obtained to form a data set for building a personalized eye movement model; the data set includes several eye movement features, and the eye movement features include gaze point coordinates, gaze duration and eye saccade frequency.

[0078] Extracting the eye movement features from the error samples and the correct operation samples and constructing an initial eye movement model;

[0079] Analyzing the eye movement features, calculating the weight coefficient of each eye movement feature to obtain a feature weight distribution;

[0080] Determining whether the feature weight distribution deviates from a preset threshold, and if so, adjusting the weight coefficient according to the user's operating habits to generate an optimized eye movement model;

[0081] Based on the optimized eye movement model, the command recognition threshold is adjusted in combination with the user's operating habits to obtain a personalized recognition model;

[0082] Processing the real-time eye movement data through the personalized recognition model to determine whether the command recognition result conforms to the user's operating habits and generate a matching result; updating the eye movement feature weight and command recognition threshold of the personalized recognition model according to the matching result;

[0083] If the matching result of the personalized recognition model does not reach the preset accuracy, the previous step is repeated to optimize the model parameters until the final personalized recognition model is obtained.

[0084] Specifically, the system continuously records user interactions with the electronic photo frame, collecting eye movement data (including correct operation samples where the user's intention is recognized and the actual feedback is consistent, or incorrect samples where it is inconsistent) to form a training dataset. Incorrect samples are those where the system's recognized control instructions do not match the user's actual intention; correct operation samples are those where the recognition results are consistent with the user's feedback.

[0085] The dataset contains multiple eye movement features, such as the gaze point coordinates, which is the specific location where the user's gaze rests, the gaze duration, which refers to the length of time the user continuously gazes at a certain area, and the eye movement frequency, which is the number of times the user's gaze quickly moves from one area to another per unit time. These features together constitute the basic representation of eye movement behavior.

[0086] The eye movement features of the two types of samples, error samples and correct operation samples, were extracted and normalized. After removing outliers, a classification algorithm was used to preliminarily model and construct a basic version of the initial eye movement model.

[0087] Methods such as information gain, mutual information, or LASSO regression are used to assess the impact of each eye movement feature on the final command recognition result, and a weight coefficient is assigned to each feature to reflect its importance in the model's judgment. For example, if a user prefers to confirm an action by gazing for a long time, the weight of "gaze duration" will be increased, while the weight of "eye saccade frequency" will be reduced accordingly. The system combines these weights into a feature weight distribution map.

[0088] A reasonable weight deviation threshold (e.g., ±0.15) is set as the criterion for determining whether the model needs adjustment. If the weight of a particular eye movement feature exceeds this threshold, it indicates that the current model does not accurately reflect the user's preferences. The weight of the corresponding feature will be dynamically adjusted based on the user's recent interaction behavior (such as undoing an action, repeating a command input, etc.), thereby generating an optimized eye movement model that better reflects user habits.

[0089] After adjusting feature weights, we further optimize the command recognition threshold. For example, when recognizing two similar commands, "turn page" and "zoom in," if a user frequently mistakenly triggers "turn page," the system will raise the threshold for triggering the "turn page" command while lowering the difficulty of recognizing the "zoom in" command, thereby better aligning with the user's actual usage habits. The model that undergoes multiple rounds of iterative optimization is the personalized recognition model.

[0090] In one embodiment, the method further comprises:

[0091] Collect user interaction data in real time through the system background; extract user identification information and historical recognition accuracy based on the user interaction data; trigger progressive learning mode if the user's recognition accuracy is lower than the preset threshold or the user is a newly registered user;

[0092] In the progressive learning mode, core functional instructions are selected from a preset instruction library to generate a simplified instruction sequence based on the user's current operating level. A corresponding visual guidance element is generated for each simplified instruction. The form and intensity of visual feedback are adjusted based on the user's eye movement characteristics to match the user's operating habits. The simplified instruction sequence covers the most basic and commonly used interactive operations.

[0093] When users use simplified instructions, eye movement and operation data are collected in real time to calculate the user's proficiency index. When the user's proficiency reaches the proficiency threshold, it is determined that the user has mastered the current instruction set.

[0094] After determining that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of new instructions.

[0095] Specifically, during operation, the system continuously collects user interaction behavior information through the built-in data collection module. This information includes but is not limited to click frequency, eye movement trajectory, command execution success rate, and response time, thereby constructing a complete user behavior profile. Based on this collected interaction data, two key parameters are extracted: user identification information (such as user ID and device binding information) that uniquely identifies the user, and the historical eye movement recognition accuracy rate, which reflects recognition performance. This refers to the proportion of time over a period of time when the system's recognition results of user intent are consistent with the actual operation.

[0096] When the system's judgment conditions are met - that is, the user's recognition accuracy is lower than the set threshold, or the current user is a newly registered user (that is, sufficient eye movement model training data has not yet been accumulated), the system will automatically enter progressive learning mode.

[0097] In this mode, the system selects the most basic and frequently used commands (such as "flip pages," "switch modes," and "zoom in") from a pre-established library of preset commands (including all available functions on the digital photo frame) based on the user's current operating level. This simplified command sequence contains only core functions, lowering the learning curve.

[0098] To help users understand and execute these simplified instructions, the system dynamically generates corresponding visual guidance elements (such as highlighted areas, arrows, and animated prompts) through a visual feedback generation algorithm. These visual elements are prominently displayed on the interface as the instructions change, guiding users to complete specific eye movements and enhancing the intuitiveness and comprehensibility of human-computer interaction.

[0099] Based on the user's real-time eye movement characteristics, their cognitive load and operational proficiency are assessed, and the form and intensity of visual guidance are dynamically adjusted accordingly to make it more in line with the user's actual operating habits and improve guidance efficiency.

[0100] While users are using simplified commands, the system will continuously collect their eye movement trajectories and operation data, and calculate the user's proficiency index based on multiple dimensions such as operation success rate, response time, and eye movement stability.

[0101] When this indicator reaches the preset proficiency threshold (such as recognition accuracy exceeding 90% for three consecutive days), the system determines that the user has mastered the current command set and has the ability to further learn complex functions. It gradually unlocks more complex commands from the preset command library, and synchronously updates the visual feedback interface, adding new guidance elements to support the learning and use of new commands.

[0102] In one embodiment, the method further comprises:

[0103] Analyze scenarios where user command execution fails to identify the error type and its cause;

[0104] Adjust the parameters of the personalized recognition model based on the distribution of error types and adjust the instruction mapping rules based on the user's operating habits.

[0105] Specifically, the system continuously records the results of user command execution during operation, and combines eye movement data, interface status information, and user feedback behavior to analyze each command execution failure and identify multiple common error types, such as: misidentification errors, omission errors, and false trigger errors. Among them, misidentification errors: the system mistakenly identifies the user's true intention as another function instruction;

[0106] Omission error: The user has made a clear operation intention but it is not captured by the system; false trigger error: The system misjudges and executes an instruction without clear user intention.

[0107] While identifying error types, we further analyze their causes, including but not limited to: unstable eye movement characteristics (such as drifting gaze points or rapid saccades), irrational command mapping (weak correlation between certain eye movement patterns and commands), interface state interference (such as current interface elements blocking key areas, affecting line of sight), and differences in user operating habits (different users have individual differences in how they express the same eye movement). Based on the analysis of these error types and their causes, we optimize the parameters of the personalized recognition model.

[0108] In one embodiment, the eyeball position is tracked in real time based on the user's eye image data, and an eye movement trajectory sequence is obtained according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state, including:

[0109] Collect user eye image data through the camera module;

[0110] Performing image preprocessing on the acquired image to obtain a first image;

[0111] Detecting the pupil area of ​​the first image based on the YOLOv5 model of deep learning to obtain the pupil bounding box coordinates and calculating the pupil center coordinates according to the bounding box center;

[0112] Detect the distance between the upper and lower eyelids to determine the eyelid opening and closing status;

[0113] Generate a time-stamped eye movement trajectory sequence based on the displacement changes of pupil center coordinates and the time series of eyelid opening and closing states in consecutive frames; wherein, when the eye movement trajectory sequence is interrupted, a re-localization mechanism is triggered to rescan the first image until stable pupil coordinates are obtained;

[0114] The trajectory sequence is smoothed using the Kalman filter algorithm to obtain a second trajectory sequence;

[0115] The Euclidean distance of pupil displacement between adjacent frames of the second trajectory sequence is detected to identify abnormal points, and the trajectory breakpoints are repaired using interpolation to form a continuous and stable third trajectory sequence;

[0116] Eye movement feature parameters are extracted based on the pupil center coordinate change rate and eyelid opening and closing frequency of the third trajectory sequence.

[0117] Specifically, the user's facial image is collected through the built-in camera module to capture image data of the eye area.

[0118] The collected original image is subjected to image preprocessing operations to remove noise interference and enhance image contrast, thereby obtaining a clear image that can be used for subsequent detection, namely the first image.

[0119] Among them, image preprocessing usually includes image enhancement methods such as grayscale, histogram equalization, and gamma correction, the purpose of which is to improve the distinguishability between the pupil and the background.

[0120] The pupil area of ​​the first image is identified using the deep learning-based target detection model YOLOv5.

[0121] The YOLOv5 model is a lightweight, high-precision target detection network that can quickly locate the pupil area in the image and output the corresponding pupil bounding box coordinates.

[0122] By calculating the geometric center point of the bounding box, the pupil center coordinates of the pupil in each frame of the image are obtained as the basic data of the eye movement trajectory.

[0123] The position of the upper and lower eyelids is detected, and the eyelid opening and closing status is determined by calculating the distance between the upper and lower eyelids.

[0124] Among them, the eyelid opening and closing state is used to assist in judging whether the user is in different eye movement behavior modes such as closing eyes, blinking or staring, and is an important part of eye movement feature extraction.

[0125] Based on the displacement changes of pupil center coordinates and the time series information of eyelid opening and closing in consecutive frames, the system generates a time-stamped eye movement trajectory sequence. The eye movement trajectory sequence refers to a data set consisting of a series of time-stamped pupil center coordinates and their corresponding eyelid states.

[0126] In actual applications, pupil recognition may fail due to factors such as lighting changes and head movement, resulting in interruptions in the eye movement trajectory. To address this, the system has a re-positioning mechanism: when an abnormal interruption in the eye movement trajectory is detected, the system will automatically trigger a rescan process, reverting to the first image to search for the pupil until stable pupil coordinates are obtained.

[0127] In order to improve the stability and accuracy of the eye movement trajectory, the system further uses the Kalman filter algorithm to smooth the original trajectory, eliminate the jitter caused by image noise or misidentification, and obtain the second trajectory sequence.

[0128] After obtaining the smoothed second trajectory sequence, the system continues to detect outliers. By calculating the Euclidean distance of pupil displacement between adjacent frames, it identifies outliers with excessive jumps and uses linear interpolation or spline interpolation methods to repair trajectory breakpoints, ultimately forming a continuous and stable third trajectory sequence.

[0129] Based on the third trajectory sequence, multiple key eye movement feature parameters are extracted by combining the change rate of pupil center coordinates, gaze dwell time, and eyelid opening and closing frequency.

[0130] In one embodiment, the gaze point coordinates, gaze duration, and eye saccade frequency are calculated based on the eye movement trajectory sequence; the user's current eye movement behavior type is obtained through a personalized recognition model; and the user's current eye movement behavior type is matched with a preset eye movement pattern library to obtain a candidate instruction set, including:

[0131] Calculate the fixation dwell time, saccade speed, and blink frequency based on the eye movement trajectory sequence:

[0132] Based on the personalized recognition model, the calculated feature vector is compared with the preset eye movement pattern library, and the K-nearest neighbor algorithm is used to calculate the similarity between the current feature vector and each pattern in the pattern library;

[0133] If the similarity between the features within a certain time period and a certain pattern in the pattern library exceeds a preset threshold, it is determined to be the corresponding eye movement behavior type;

[0134] For the identified eye movement behavior type, a set of candidate instruction sets associated with it is filtered out from the preset instruction library through mapping relationships.

[0135] In one embodiment, if there are multiple candidate instructions in the candidate instruction set, the user's historical operation preferences and current interface state information are analyzed to prioritize the candidate instructions; the candidate instruction with the highest priority is determined as the user's actual operation intention and a corresponding control instruction is generated, including:

[0136] Obtain user historical operation data and current interface status information;

[0137] Standardize the collected operation data and interface status information to form an input data set;

[0138] Using the Bayesian probability model to calculate the initial probability distribution of each candidate instruction based on the standardized input data set to obtain the probability value of each instruction;

[0139] Prioritizing the candidate instructions using a weighted summation method according to preset historical preference weights and interface state relevance weights to generate a sorted instruction priority list;

[0140] If the probability value of the highest priority instruction in the instruction priority list exceeds the preset probability value, it is regarded as the user's actual operation intention;

[0141] Generate corresponding control instructions based on the determined user's actual operation intention.

[0142] In one embodiment, a photo frame function operation is executed according to the control instruction; and a user's subsequent eye movement behavior is monitored to determine whether the result of the executed photo frame function operation meets expectations. If an undo eye movement behavior is identified after the photo frame function operation is executed, the executed photo frame function operation is canceled and the corresponding control instruction is marked as an error sample, including:

[0143] Get control instructions and call the photo frame function module to perform corresponding operations and update the interface status; synchronously record operation logs, which include user ID, instruction content, execution status and timestamp information;

[0144] After executing the control command, the eye tracking device is started to collect the user's gaze point coordinate sequence in real time; the eye movement data within a preset time period after the operation is extracted for analysis;

[0145] Cluster analysis is performed on the collected eye movement data, and the K-means algorithm is used to divide the gaze area. The Euclidean distance between the current gaze center and the preset undo area in the interface is calculated. If the gaze center is within the undo area and the gaze duration exceeds the preset duration, it is determined to be an undo-type eye movement behavior.

[0146] If an undo eye movement behavior is identified, it is determined that the execution of this control instruction does not meet the user's expectations; the executed photo frame function operation is undoed according to the instruction identifier in the operation log;

[0147] The control instruction and its associated eye movement behavior data are marked as error samples and stored in the error sample database.

[0148] In one embodiment, the method further comprises:

[0149] Obtaining the user's most recent several operation sequences; using the most recent several operation sequences as context input;

[0150] Initialize the long short-term memory neural network model;

[0151] The most recent several operation sequences are input into the LSTM model; the LSTM model learns and extracts the temporal dependencies between operations and outputs a vector representing the contextual features of the current operation sequence; the vector represents the historical pattern of user operations and their temporal correlation.

[0152] Get the status information of the current interface in real time;

[0153] The operation context feature vector output by the LSTM model is concatenated with the current interface state information to form a new comprehensive feature vector.

[0154] The new comprehensive feature vector is input into the Bayesian probability model to calculate the probability of the candidate instruction.

[0155] Specifically, the system obtains the user's most recent operation sequences and uses them as contextual input to feed a pre-initialized long short-term memory (LSTM) neural network model. This contextual input refers to historical data used to model the temporal continuity of user operation behaviors. The LSTM model is a deep learning model specialized for processing time series data. It has the ability to memorize long-term dependencies and automatically extract the temporal features and underlying patterns between user operations.

[0156] Through training and inference, the LSTM model learns and extracts temporal dependencies in operation sequences, outputting a feature vector representing the contextual characteristics of the current operation. A feature vector is a high-dimensional numerical vector that characterizes the historical patterns of a user's recent operations and their temporal correlation, such as a tendency to perform the same type of operations consecutively or a preference for a specific operation order.

[0157] Collect the status information of the current interface, including but not limited to: the functional module currently in which the device is located, the location and layout of each functional area on the interface, whether there is an unexecuted operation queue, and the current display content.

[0158] In order to achieve joint modeling of context information and interface status, the system splices the operation context feature vector output by the LSTM model with the current interface status information to form a new comprehensive feature vector.

[0159] The comprehensive feature vector is input into the Bayesian probability model to calculate the probability of the candidate instructions.

[0160] Through the above method, the system can not only identify the possible instructions corresponding to the current eye movement behavior, but also combine the user's recent operating habits and interface status changes to effectively improve the recognition accuracy of ambiguous or multi-meaning eye movement behaviors.

[0161] Reference Figure 2 The second embodiment of the present invention provides an electronic photo frame intelligent interactive control system based on eye tracking, comprising:

[0162] The first processing module is used to: track the eyeball position in real time based on the user's eye image data, and obtain an eye movement trajectory sequence according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state;

[0163] The second processing module is configured to: calculate the gaze point coordinates, gaze duration, and eye saccade frequency based on the eye movement trajectory sequence; obtain the user's current eye movement behavior type through a personalized recognition model; and match the user's current eye movement behavior type with a preset eye movement pattern library to obtain a candidate instruction set;

[0164] A third processing module is configured to: if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface state information to prioritize the candidate instructions; determine the candidate instruction with the highest priority as the user's actual operation intention and generate a corresponding control instruction;

[0165] The fourth processing module is used to: execute a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo-type eye movement behavior is identified after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0166] It should be noted that an embodiment of the present invention provides an electronic photo frame intelligent interactive control system based on eye tracking, which is used to execute all the process steps of an electronic photo frame intelligent interactive control method based on eye tracking in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and therefore will not be described in detail.

[0167] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An intelligent interactive control method for an electronic photo frame based on eye tracking, characterized in that: include: The eyeball position is tracked in real time based on the user's eye image data, and the eye movement trajectory sequence is obtained according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state; Calculating the gaze point coordinates, gaze duration and saccade frequency based on the eye movement trajectory sequence; And obtain the user's current eye movement behavior type through personalized recognition model; Based on the user's current eye movement behavior type, the user matches the preset eye movement pattern library and obtains a set of candidate instructions; If there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface status information to prioritize the candidate instructions; Determine the candidate instruction with the highest priority as the user's actual operation intention and generate a corresponding control instruction; Executing a photo frame function operation according to the control instruction; monitoring the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo eye movement behavior is identified after the photo frame function operation is executed, canceling the executed photo frame function operation and marking the corresponding control instruction as an error sample; The error samples are used to update the model parameters of the personalized recognition model; The steps of constructing the personalized recognition model include: Obtaining error samples and correct operation samples of the user to form a data set for building a personalized eye movement model; the data set includes a number of eye movement features, the eye movement features including gaze point coordinates, gaze duration, and eye saccade frequency; Extracting the eye movement features from the error samples and the correct operation samples and constructing an initial eye movement model; Analyzing the eye movement features, calculating the weight coefficient of each eye movement feature to obtain a feature weight distribution; Determining whether the feature weight distribution deviates from a preset threshold, and if so, adjusting the weight coefficient according to the user's operating habits to generate an optimized eye movement model; Based on the optimized eye movement model, the command recognition threshold is adjusted in combination with the user's operating habits to obtain a personalized recognition model; Processing the real-time eye movement data through the personalized recognition model to determine whether the command recognition result conforms to the user's operating habits and generate a matching result; updating the eye movement feature weight and command recognition threshold of the personalized recognition model according to the matching result; If the matching result of the personalized recognition model does not reach the preset accuracy, the previous step is repeated to optimize the model parameters until the final personalized recognition model is obtained.

2. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 1, characterized in that: The method further comprises: Collect user interaction data in real time through the system background; extract user identification information and historical recognition accuracy based on the user interaction data; trigger progressive learning mode if the user's recognition accuracy is lower than the preset threshold or the user is a newly registered user; In the progressive learning mode, core functional instructions are selected from a preset instruction library to generate a simplified instruction sequence based on the user's current operating level. A corresponding visual guidance element is generated for each simplified instruction. The form and intensity of visual feedback are adjusted based on the user's eye movement characteristics to match the user's operating habits. The simplified instruction sequence covers the most basic and commonly used interactive operations. When users use simplified instructions, eye movement and operation data are collected in real time to calculate the user's proficiency index. When the user's proficiency reaches the proficiency threshold, it is determined that the user has mastered the current instruction set. After determining that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of new instructions.

3. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 2, characterized in that: The method further comprises: Analyze scenarios where user command execution fails to identify the error type and its cause; Adjust the parameters of the personalized recognition model based on the distribution of error types and adjust the instruction mapping rules based on the user's operating habits.

4. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 3, characterized in that: The eye position is tracked in real time based on the user's eye image data. The eye movement trajectory sequence is obtained according to the changes in pupil center coordinates and eyelid opening and closing states, including: Collect user eye image data through the camera module; Performing image preprocessing on the acquired image to obtain a first image; Detecting the pupil area of ​​the first image based on the YOLOv5 model of deep learning to obtain the pupil bounding box coordinates and calculating the pupil center coordinates according to the bounding box center; Detect the distance between the upper and lower eyelids to determine the eyelid opening and closing status; Generate a time-stamped eye movement trajectory sequence based on the displacement changes of pupil center coordinates and the time series of eyelid opening and closing states in consecutive frames; wherein, when the eye movement trajectory sequence is interrupted, a re-localization mechanism is triggered to rescan the first image until stable pupil coordinates are obtained; The trajectory sequence is smoothed using the Kalman filter algorithm to obtain a second trajectory sequence; The Euclidean distance of pupil displacement between adjacent frames of the second trajectory sequence is detected to identify abnormal points, and the trajectory breakpoints are repaired using interpolation to form a continuous and stable third trajectory sequence; Eye movement feature parameters are extracted based on the pupil center coordinate change rate and eyelid opening and closing frequency of the third trajectory sequence.

5. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 4, characterized in that: Calculating the gaze point coordinates, gaze duration and saccade frequency based on the eye movement trajectory sequence; And obtain the user's current eye movement behavior type through personalized recognition model; Based on the user's current eye movement behavior type, the preset eye movement pattern library is matched to obtain a set of candidate instructions, including: Calculate the fixation dwell time, saccade speed, and blink frequency based on the eye movement trajectory sequence: Based on the personalized recognition model, the calculated feature vector is compared with the preset eye movement pattern library, and the K-nearest neighbor algorithm is used to calculate the similarity between the current feature vector and each pattern in the pattern library; If the similarity between the features within a certain time period and a certain pattern in the pattern library exceeds a preset threshold, it is determined to be the corresponding eye movement behavior type; For the identified eye movement behavior type, a set of candidate instruction sets associated with it is filtered out from the preset instruction library through mapping relationships.

6. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 5, characterized in that: If there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface status information to prioritize the candidate instructions; The candidate instruction ranked first in priority is determined as the user's actual operation intention and the corresponding control instruction is generated, including: Obtain user historical operation data and current interface status information; Standardize the collected operation data and interface status information to form an input data set; Using the Bayesian probability model to calculate the initial probability distribution of each candidate instruction based on the standardized input data set to obtain the probability value of each instruction; Prioritizing the candidate instructions using a weighted summation method according to preset historical preference weights and interface state relevance weights to generate a sorted instruction priority list; If the probability value of the highest priority instruction in the instruction priority list exceeds the preset probability value, it is regarded as the user's actual operation intention; Generate corresponding control instructions based on the determined user's actual operation intention.

7. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 6, characterized in that: Executing a photo frame function operation according to the control instruction; monitoring the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo eye movement behavior is identified after the photo frame function operation is executed, canceling the executed photo frame function operation and marking the corresponding control instruction as an error sample, including: Get control instructions and call the photo frame function module to perform corresponding operations and update the interface status; synchronously record operation logs, which include user ID, instruction content, execution status and timestamp information; After executing the control command, the eye tracking device is started to collect the user's gaze point coordinate sequence in real time; the eye movement data within a preset time period after the operation is extracted for analysis; Cluster analysis is performed on the collected eye movement data, and the K-means algorithm is used to divide the gaze area. The Euclidean distance between the current gaze center and the preset undo area in the interface is calculated. If the gaze center is within the undo area and the gaze duration exceeds the preset duration, it is determined to be an undo-type eye movement behavior. If an undo eye movement behavior is identified, it is determined that the execution of this control instruction does not meet the user's expectations; the executed photo frame function operation is undoed according to the instruction identifier in the operation log; The control instruction and its associated eye movement behavior data are marked as error samples and stored in the error sample database.

8. The method for intelligent interactive control of an electronic photo frame based on eye tracking according to claim 7, characterized in that: The method further comprises: Obtaining the user's most recent several operation sequences; using the most recent several operation sequences as context input; Initialize the long short-term memory neural network model; Input the most recent several operation sequences into the LSTM model; the LSTM model learns and extracts the temporal dependencies between operations and outputs a vector representing the contextual features of the current operation sequence; the vector represents the historical pattern of user operations and their temporal correlation; Get the status information of the current interface in real time; The operation context feature vector output by the LSTM model is concatenated with the current interface state information to form a new comprehensive feature vector. The new comprehensive feature vector is input into the Bayesian probability model to calculate the probability of the candidate instruction.

9. An electronic photo frame intelligent interactive control system based on eye tracking, used to execute the method according to any one of claims 1 to 8, characterized in that: include: The first processing module is used to: track the eyeball position in real time based on the user's eye image data, and obtain an eye movement trajectory sequence according to the changes in the pupil center coordinates and the changes in the eyelid opening and closing state; A second processing module is configured to calculate the gaze point coordinates, gaze duration, and saccade frequency based on the eye movement trajectory sequence; And obtain the user's current eye movement behavior type through personalized recognition model; Based on the user's current eye movement behavior type, the user matches the preset eye movement pattern library and obtains a set of candidate instructions; A third processing module is configured to: if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and current interface state information to prioritize the candidate instructions; determine the candidate instruction with the highest priority as the user's actual operation intention and generate a corresponding control instruction; The fourth processing module is used to: execute a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets expectations; if an undo-type eye movement behavior is identified after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

Images