Electronic photo frame intelligent interaction control method and system based on eye movement tracking

Abstract

The invention relates to the field of electronic photo frames, and discloses an electronic photo frame intelligent interaction control method and system based on eye movement tracking, and the method comprises the steps: carrying out the real-time tracking of an eyeball position based on the eye image data of a user, and obtaining an eye movement track sequence according to the pupil center coordinate change and the eyelid opening and closing state change; calculating a fixation point coordinate, a fixation time length and an eye hopping frequency based on the eye movement track sequence; the current eye movement behavior type of the user is obtained through a personalized recognition model; matching with a preset eye movement pattern library based on the current eye movement behavior type of the user to obtain a candidate instruction set; and executing photo frame function operation according to the control instruction. The method has the following effect that the convenience and accuracy of man-machine interaction are improved based on non-contact intelligent control of eye movement.

Description

Technical Field

[0001] The present invention relates to the field of electronic photo frames, and particularly to an intelligent interactive control method and system for an electronic photo frame based on eye tracking. Background Art

[0002] At present, with the rapid development of human-computer interaction technology, the interaction methods of intelligent display devices are undergoing profound changes. As an emerging non-contact interaction technology, eye tracking shows great application potential and market value in consumer electronic products such as electronic photo frames.

[0003] Most of the current electronic photo frames on the market adopt traditional interaction methods such as touch screens, physical buttons or remote controls. These methods have obvious shortcomings in specific scenarios. For example, touch operations are likely to leave fingerprints on the screen, affecting the viewing effect; physical buttons not only increase the structural complexity of the device but also increase the probability of failures; remote controls are prone to being lost and require additional storage space.

[0004] When applying eye tracking technology to the intelligent interactive control of electronic photo frames, there are many technical problems in the design and recognition of eye movement instructions. There are significant individual differences in users' eye movement behaviors, and they will change with the usage scenarios. The same eye movement action may express completely different operation intentions for different users in different states. This uncertainty in intention recognition directly affects the system's accurate judgment of the user's true operation intention.

[0005] How to solve the above technical problems is a technical challenge that those skilled in the art need to overcome. Summary of the Invention

[0006] The present invention provides an intelligent interactive control method and system for an electronic photo frame based on eye tracking to at least partially solve the above technical problems.

[0007] In a first aspect, to solve the above technical problems, the present invention provides an intelligent interactive control method for an electronic photo frame based on eye tracking, including: Real-time tracking of the eye position based on the user's eye image data, and obtaining an eye movement trajectory sequence according to the change of the pupil center coordinates and the change of the eyelid opening and closing state; Calculating the fixation point coordinates, fixation duration and saccade frequency based on the eye movement trajectory sequence; and obtaining the current eye movement behavior type of the user through a personalized recognition model; matching the user's current eye movement behavior type with a preset eye movement pattern library to obtain a candidate instruction set; If there are multiple candidate instructions in the candidate instruction set, analyzing the user's historical operation preferences and the current interface state information to rank the priority of each candidate instruction; determining the candidate instruction with the highest priority as the user's true operation intention and generating a corresponding control instruction; Execute the photo frame function operation according to the control instruction; monitor the subsequent eye movement behavior of the user to determine whether the result of the executed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the execution of the photo frame function operation, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0008] In an alternative embodiment, the steps for constructing the personalized recognition model include: Obtain the error samples and correct operation samples of the user to form a data set for constructing a personalized eye movement model; the data set includes several eye movement features, and the eye movement features include fixation point coordinates, fixation duration, and saccade frequency; Extract the eye movement features from the error samples and correct operation samples and construct an initial eye movement model; Analyze the eye movement features, calculate the weight coefficients of each eye movement feature to obtain a feature weight distribution; Determine whether the feature weight distribution deviates from a preset threshold. If it deviates, adjust the weight coefficients according to the user's operation habits to generate an optimized eye movement model; Based on the optimized eye movement model, adjust the instruction recognition threshold in combination with the user's operation habits to obtain a personalized recognition model; Process the real-time eye movement data through the personalized recognition model, judge whether the instruction recognition result conforms to the user's operation habits to generate a matching result; update the eye movement feature weights and instruction recognition thresholds of the personalized recognition model according to the matching result; If the matching result of the personalized recognition model does not reach the preset accuracy rate, repeat the above steps to optimize the model parameters until the final personalized recognition model is obtained.

[0009] In an alternative embodiment, the method further includes: Collect the user's interaction data in real time through the system background; extract the user identification information and historical recognition accuracy rate based on the user's interaction data; if the user's recognition accuracy rate is lower than the preset threshold or the user is a newly registered user, trigger the progressive learning mode; In the progressive learning mode, according to the user's current operation level, screen out the core function instructions from the preset instruction library to generate a simplified instruction sequence; generate corresponding visual guidance elements for each simplified instruction; adjust the form and intensity of the visual feedback based on the user's eye movement features to match the user's operation habits; the simplified instruction sequence covers the most basic and commonly used interaction operations; Collect the eye movement trajectory and operation data in real time during the user's use of the simplified instructions, calculate the user proficiency index; when the user's proficiency reaches the proficiency threshold, it is determined that the user has mastered the current instruction set; After it is determined that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of the new instructions.

[0010] In an alternative embodiment, the method further includes: Analyze the scenarios where the user's instruction execution fails to identify the error type and its cause; Adjust the parameters of the personalized recognition model based on the error type distribution and adjust the instruction mapping rules in combination with the user's operation habits.

[0011] In an alternative embodiment, the real-time tracking of the eyeball position is performed based on the user's eye image data, and an eye movement trajectory sequence is obtained according to the changes in the pupil center coordinates and the eyelid opening and closing states, including: Collect the user's eye image data through the camera module; Perform image preprocessing on the collected image to obtain a first image; Based on the YOLOv5 model of deep learning, detect the pupil area of the first image to obtain the pupil bounding box coordinates and calculate the pupil center coordinates according to the center of the bounding box; Detect the distance between the upper and lower eyelids to determine the eyelid opening and closing state; Generate a timestamped eye movement trajectory sequence according to the displacement change of the pupil center coordinates and the time series of the eyelid opening and closing states in consecutive frames; wherein, when the eye movement trajectory sequence is interrupted, trigger a repositioning mechanism to rescan the first image until stable pupil coordinates are obtained; Use the Kalman filter algorithm to smooth the trajectory sequence to obtain a second trajectory sequence; Detect the Euclidean distance of the pupil displacement between adjacent frames of the second trajectory sequence to identify abnormal points, and use the interpolation method to repair the trajectory breakpoints to form a continuous and stable third trajectory sequence; Extract eye movement feature parameters according to the change rate of the pupil center coordinates and the eyelid opening and closing frequency of the third trajectory sequence.

[0012] In an alternative embodiment, calculate the fixation point coordinates, fixation duration, and saccade frequency based on the eye movement trajectory sequence; and obtain the user's current eye movement behavior type through the personalized recognition model; match the user's current eye movement behavior type with the preset eye movement pattern library to obtain a candidate instruction set, including: Calculate the fixation point residence time, saccade speed, and detect the blink frequency according to the eye movement trajectory sequence: Based on the personalized recognition model, compare the calculated feature vector with the preset eye movement pattern library, and use the K-nearest neighbor algorithm to calculate the similarity between the current feature vector and each pattern in the pattern library; If the similarity between the features within a certain time period and a certain pattern in the pattern library exceeds a preset threshold, it is determined as the corresponding eye movement behavior type; For the identified eye movement behavior type, a set of candidate instruction sets associated with it are filtered out from the preset instruction library through the mapping relationship.

[0013] In an optional implementation manner, if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and the current interface state information to rank the priority of each candidate instruction; determine the candidate instruction with the highest priority as the user's true operation intention and generate the corresponding control instruction, including: Obtain the user's historical operation data and the current interface state information; Perform standardization processing on the collected operation data and interface state information to form an input data set; Use the Bayesian probability model to calculate the initial probability distribution of each candidate instruction based on the standardized input data set to obtain the probability values of each instruction; Use the weighted summation method to rank the priority of the candidate instructions according to the preset historical preference weight and interface state correlation weight to generate a sorted instruction priority list; If the probability value of the instruction with the highest priority in the instruction priority list exceeds the preset probability value, then use it as the user's true operation intention; Generate the corresponding control instruction based on the determined user's true operation intention.

[0014] In an optional implementation manner, perform the photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample, including: Obtain the control instruction and call the photo frame function module to perform the corresponding operation and update the interface state; synchronously record the operation log, and the operation log includes user ID, instruction content, execution status, and timestamp information; After executing the control instruction, start the eye movement tracking device to collect the user's fixation point coordinate sequence in real time; extract the eye movement data within a preset time period after the operation for analysis; Perform clustering analysis on the collected eye movement data, use the K-means algorithm to divide the fixation area; calculate the Euclidean distance between the current fixation center and the preset cancellation area in the interface; if the fixation center is within the cancellation area and the fixation duration exceeds the preset duration, it is determined as a cancellation-type eye movement behavior; If a cancellation-type eye movement behavior is recognized, it is determined that the execution of this control instruction does not meet the user's expectation; cancel the executed photo frame function operation according to the instruction identifier in the operation log; Mark the control instruction and its associated eye movement behavior data as error samples and store them in the error sample database.

[0015] In an alternative embodiment, the method further includes: Obtain the recent several operation sequences of the user; use the recent several operation sequences as context inputs; Initialize the long short-term memory neural network model; Input the recent several operation sequences into the LSTM model; the LSTM model learns and extracts the temporal dependence relationships between operations and outputs a vector representing the context features of the current operation sequence; the vector represents the historical pattern of the user's operations and their temporal correlations.

[0016] Obtain the status information of the current interface in real time; Concatenate the operation context feature vector output by the LSTM model with the current interface status information to form a new comprehensive feature vector; Input the new comprehensive feature vector into the Bayesian probability model to calculate the candidate instruction probabilities.

[0017] In a second aspect, the present invention provides an intelligent interactive control system for an electronic photo frame based on eye movement tracking, including: A first processing module for: performing real-time tracking of the eyeball position based on the user's eye image data, and obtaining an eye movement trajectory sequence according to the changes in the pupil center coordinates and the eyelid opening and closing states; A second processing module for: calculating the fixation point coordinates, fixation duration, and saccade frequency based on the eye movement trajectory sequence; and obtaining the current eye movement behavior type of the user through a personalized recognition model; matching the current eye movement behavior type of the user with a preset eye movement pattern library to obtain a candidate instruction set; A third processing module for: if there are multiple candidate instructions in the candidate instruction set, analyzing the user's historical operation preferences and the current interface status information to perform priority ranking on each candidate instruction; determining the candidate instruction with the first priority ranking as the user's true operation intention and generating a corresponding control instruction; A fourth processing module for: performing a photo frame function operation according to the control instruction; monitoring the user's subsequent eye movement behavior to determine whether the result of the performed photo frame function operation meets the expectation, and if a cancellation-type eye movement behavior is recognized after the photo frame function operation is performed, canceling the performed photo frame function operation and marking the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: contactless intelligent control based on eye movement improves the convenience and accuracy of the human-computer interaction of the electronic photo frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flowchart of a method for intelligent interactive control of an electronic photo frame based on eye movement tracking provided by the first embodiment of the present invention; Figure 2 It is a block diagram of a system for intelligent interactive control of an electronic photo frame based on eye movement tracking provided by the second embodiment of the present invention. Specific embodiments

[0020] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0021] Referring to Figure 1 , the first embodiment of the present invention provides a method for intelligent interactive control of an electronic photo frame based on eye movement tracking, including the following steps: S101, based on the user's eye image data, the position of the eyeball is tracked in real time, and an eye movement trajectory sequence is obtained according to the change of the pupil center coordinates and the change of the eyelid opening and closing state; S102, based on the eye movement trajectory sequence, calculate the fixation point coordinates, fixation duration, and saccade frequency; and obtain the current eye movement behavior type of the user through a personalized recognition model; match the current eye movement behavior type of the user with a preset eye movement pattern library to obtain a candidate instruction set; S103, if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and the current interface state information to sort the priorities of each candidate instruction; determine the candidate instruction with the highest priority as the user's true operation intention and generate a corresponding control instruction; S104, perform the photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0022] In one implementation, the construction steps of the personalized recognition model include: Obtain the user's error samples and correct operation samples to form a data set for constructing a personalized eye movement model; the data set includes several eye movement features, and the eye movement features include fixation point coordinates, fixation duration, and saccade frequency.

[0023] Extract the eye movement features from the error samples and correct operation samples and construct an initial eye movement model; Analyze the eye movement features, calculate the weight coefficients of each eye movement feature to obtain a feature weight distribution; Determine whether the feature weight distribution deviates from a preset threshold. If it deviates, adjust the weight coefficient according to the user's operation habit to generate an optimized eye movement model; Based on the optimized eye movement model, adjust the instruction recognition threshold in combination with the user's operation habit to obtain a personalized recognition model; Process the real-time eye movement data through the personalized recognition model, determine whether the instruction recognition result conforms to the user's operation habit to generate a matching result; update the eye movement feature weights and instruction recognition threshold of the personalized recognition model according to the matching result; If the matching result of the personalized recognition model does not reach the preset accuracy rate, repeat the above steps to optimize the model parameters until the final personalized recognition model is obtained.

[0024] Specifically, the system continuously records the interaction behavior between the user and the digital photo frame, collects eye movement data (including correct operation samples where the user intention recognition result is consistent with the actual feedback or error samples where they are inconsistent). An error sample refers to a sample where the control instruction recognized by the system does not match the user's true intention; a correct operation sample refers to a sample where the recognition result is consistent with the user's feedback.

[0025] The dataset contains multiple eye movement features, such as the fixation point coordinates, which is the specific position where the user's line of sight stays, the fixation duration, which refers to the length of time the user continuously fixates on a certain area, and the saccade frequency, which is the number of times the user's line of sight quickly moves from one area to another within a unit of time. These features together constitute the basic representation of eye movement behavior.

[0026] Extract and normalize the eye movement features in the above two types of samples, namely error samples and correct operation samples. After removing outliers, use a classification algorithm to initially model and construct a basic version of the initial eye movement model.

[0027] Adopt methods such as information gain, mutual information method or LASSO regression to evaluate the influence degree of each eye movement feature on the final instruction recognition result, and assign a weight coefficient to each feature to reflect its importance in model judgment. For example, if a certain user is more inclined to confirm an operation by long-term fixation, the weight of "fixation duration" will be increased, while the weight of "saccade frequency" will be correspondingly decreased. The system combines these weights into a feature weight distribution map.

[0028] Set a reasonable weight deviation threshold (e.g., ±0.15) as the criterion for judging whether the model needs to be adjusted. If it is found that the weight of a certain eye movement feature exceeds this threshold, it indicates that the current model fails to accurately reflect the user's operation preferences. The weights of the corresponding features will be dynamically adjusted in combination with the user's latest interaction behaviors (such as undo operations, repeated instruction inputs, etc.) to generate an optimized eye movement model that is closer to the user's habits.

[0029] On the basis of completing the adjustment of feature weights, further optimize the instruction recognition threshold. For example, when recognizing two similar instructions, "page turning" and "zooming in", if the user often accidentally touches "page turning", the system will increase the trigger threshold of the "page turning" instruction and at the same time reduce the recognition difficulty of the "zooming in" instruction, so as to better conform to the user's actual usage habits. The model after multiple rounds of iterative optimization is the personalized recognition model.

[0030] In one implementation, the method further includes: Real-time collect the user's interaction data through the system background; extract the user identification information and historical recognition accuracy based on the user's interaction data; trigger the progressive learning mode if the user's recognition accuracy is lower than the preset threshold or the user is a newly registered user; In the progressive learning mode, according to the user's current operation level, screen out the core function instructions from the preset instruction library to generate a simplified instruction sequence; generate corresponding visual guidance elements for each simplified instruction; adjust the form and intensity of visual feedback based on the user's eye movement features to match the user's operation habits; the simplified instruction sequence covers the most basic and commonly used interaction operations; During the user's use of the simplified instructions, real-time collect the eye movement trajectory and operation data, and calculate the user proficiency index; when the user's proficiency reaches the proficiency threshold, it is determined that the current instruction set has been mastered; After it is determined that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of new instructions.

[0031] Specifically, during the operation of the system, continuously obtain the user's interaction behavior information through the built-in data collection module. The interaction behavior information includes but is not limited to click frequency, eye movement trajectory, instruction execution success rate, response time, so as to construct a complete user behavior portrait. Based on the collected interaction data, two key parameters are extracted: one is the user identification information (such as user ID, device binding information, etc.) used to uniquely identify the user's identity, and the other is the historical eye movement recognition accuracy reflecting the recognition performance, that is, the ratio of the system's recognition results of the user's intention to the actual operation within a certain period of time in the past.

[0032] When the determination condition of the system is met - that is, the user's recognition accuracy is lower than the set threshold, or the current user is a newly registered user (i.e., not enough eye movement model training data has been accumulated), the system will automatically enter the progressive learning mode.

[0033] In this mode, the system will screen out the most basic and frequently used function instructions (such as "page turning", "mode switching", "zooming in") from the pre-established preset instruction library (including all available function instructions of the digital photo frame) according to the user's current operation level, and generate a simplified instruction sequence. The simplified instruction sequence only contains core function instructions, reducing the user's learning threshold.

[0034] To assist the user in understanding and executing these simplified instructions, the system dynamically generates corresponding visual guidance elements (such as highlighted areas, arrow indications, animation prompts) through a visual feedback generation algorithm. These visual elements will be presented prominently on the interface as the instructions change, guiding the user to complete specific eye movement operations and enhancing the intuitiveness and comprehensibility of human-computer interaction.

[0035] Based on the user's real-time eye movement characteristics, evaluate their cognitive load and operation proficiency, and accordingly dynamically adjust the form and intensity of visual guidance to make it more in line with the user's actual operation habits and improve the guidance efficiency.

[0036] During the user's use of the simplified instructions, the system will continuously collect their eye movement trajectories and operation data, and calculate the user's proficiency index in multiple dimensions such as operation success rate, response time, and eye movement stability.

[0037] When this index reaches the preset proficiency threshold (such as the recognition accuracy exceeding 90% for three consecutive days), the system determines that the user has mastered the current instruction set and has the ability to further learn complex functions, gradually unlocks more complex instructions from the preset instruction library, and synchronously updates the visual feedback interface to add new guidance elements to support the learning and use of the new instructions.

[0038] In one implementation, the method further includes: Analyze the scenarios where the user's instruction execution fails to identify the error type and its occurrence reason; Based on the error type distribution, adjust the parameters of the personalized recognition model and adjust the instruction mapping rules in combination with the user's operation habits.

[0039] Specifically, the system continuously records the user's instruction execution results during operation, and combines the eye movement trajectory data, interface state information, and user feedback behavior to analyze each instruction execution failure, identifying various common error types, such as: misrecognition errors, omission errors, and mis-triggering errors. Among them, misrecognition errors: the system misidentifies the user's true intention as other function instructions; Omission errors: The user has a clear operation intention but it is not captured by the system; False trigger errors: The system misjudges and executes a certain instruction without a clear user intention.

[0040] While identifying the error type, further analyze its causes, including but not limited to: unstable eye movement features (such as fixation point drift, too fast saccade speed), unreasonable instruction mapping (the correlation between certain eye movement behavior patterns and instructions is weak), interface state interference (such as current interface elements blocking key areas, affecting line of sight judgment), and differences in user operation habits (different users have individual differences in the expression of the same eye movement behavior). Based on the analysis results of the above error types and their causes, optimize the parameters of the personalized recognition model.

[0041] In one implementation, based on the user's eye image data, the eye position is tracked in real time, and an eye movement trajectory sequence is obtained according to the change of the pupil center coordinates and the change of the eyelid opening and closing state, including: Collect the user's eye image data through the camera module; Perform image preprocessing on the collected image to obtain the first image; Based on the YOLOv5 model of deep learning, detect the pupil area of the first image to obtain the pupil bounding box coordinates and calculate the pupil center coordinates according to the center of the bounding box; Detect the distance between the upper and lower eyelids to judge the eyelid opening and closing state; Generate a timestamped eye movement trajectory sequence according to the displacement change of the pupil center coordinates and the time series of the eyelid opening and closing state in consecutive frames; among them, when the eye movement trajectory sequence is interrupted, trigger the repositioning mechanism to rescan the first image until stable pupil coordinates are obtained; Use the Kalman filter algorithm to smooth the trajectory sequence to obtain the second trajectory sequence; Detect the Euclidean distance of the pupil displacement between adjacent frames of the second trajectory sequence to identify abnormal points, and use the interpolation method to repair the trajectory breakpoints to form a continuous and stable third trajectory sequence; Extract eye movement feature parameters according to the change rate of the pupil center coordinates and the eyelid opening and closing frequency of the third trajectory sequence.

[0042] Specifically, collect the user's facial image through the built-in camera module and capture the image data of the eye area.

[0043] Perform image preprocessing operations on the collected original image to remove noise interference and enhance the image contrast, so as to obtain a clear image for subsequent detection - that is, the first image.

[0044] Among them, image preprocessing usually includes image enhancement means such as grayscale conversion, histogram equalization, and gamma correction, aiming to improve the distinguishability between the pupil and the background.

[0045] Use the YOLOv5 object detection model based on deep learning to identify the pupil area in the first image.

[0046] Among them, the YOLOv5 model is a lightweight and high-precision object detection network, which can quickly locate the pupil area in the image and output the corresponding pupil bounding box coordinates.

[0047] By calculating the geometric center point of the bounding box, the pupil center coordinates of the pupil in each frame of the image are obtained as the basic data of the eye movement trajectory.

[0048] Detect the positions of the upper and lower eyelids, and judge the eyelid opening and closing state by calculating the distance between the upper and lower eyelids.

[0049] Among them, the eyelid opening and closing state is used to assist in judging whether the user is in different eye movement behavior modes such as closing eyes, blinking, or gazing, and is an important part of eye movement feature extraction.

[0050] Based on the displacement change of the pupil center coordinates in consecutive frames and the time series information of the eyelid opening and closing state, the system generates an eye movement trajectory sequence with timestamps. Among them, the eye movement trajectory sequence refers to a data set composed of a series of pupil center coordinates with time marks and their corresponding eyelid states.

[0051] In the actual application process, pupil recognition may fail due to reasons such as light changes and head movements, resulting in the interruption of the eye movement trajectory. For this reason, the system is equipped with a repositioning mechanism: when an abnormal interruption of the eye movement trajectory is detected, the system will automatically trigger a rescan process, roll back to the first image for pupil search until stable pupil coordinates are obtained.

[0052] In order to improve the stability and accuracy of the eye movement trajectory, the system further uses the Kalman filter algorithm to smooth the original trajectory, eliminate the jitter caused by image noise or misrecognition, and obtain the second trajectory sequence.

[0053] After obtaining the smoothed second trajectory sequence, the system continues to perform outlier detection on it, identifies outliers with too large jumps by calculating the Euclidean distance of pupil displacement between adjacent frames, and uses linear interpolation or spline interpolation methods to repair the trajectory breakpoints, and finally forms a continuous and stable third trajectory sequence.

[0054] Based on this third trajectory sequence, combined with the change rate of the pupil center coordinates, the fixation dwell time, and the eyelid opening and closing frequency, multiple key eye movement feature parameters are extracted.

[0055] In one implementation, the fixation point coordinates, fixation duration, and saccade frequency are calculated based on the eye movement trajectory sequence; and the current eye movement behavior type of the user is obtained through a personalized recognition model; the current eye movement behavior type of the user is matched with a preset eye movement pattern library to obtain a candidate instruction set, including: Calculate the fixation duration, saccade speed, and detect the blink frequency based on the eye movement trajectory sequence: Based on the personalized recognition model, the calculated feature vector is compared with the preset eye movement pattern library, and the K-nearest neighbor algorithm is used to calculate the similarity between the current feature vector and each pattern in the pattern library; If the similarity of the features in a certain time period to a certain pattern in the pattern library exceeds the preset threshold, it is determined as the corresponding eye movement behavior type; For the identified eye movement behavior type, a set of candidate instruction sets associated with it are filtered out from the preset instruction library through a mapping relationship.

[0056] In one implementation, if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and the current interface state information to prioritize each candidate instruction; determine the candidate instruction with the highest priority as the user's true operation intention and generate a corresponding control instruction, including: Obtain the user's historical operation data and the current interface state information; Standardize the collected operation data and interface state information to form an input data set; Use the Bayesian probability model to calculate the initial probability distribution of each candidate instruction based on the standardized input data set to obtain the probability values of each instruction; Use the weighted summation method to prioritize the candidate instructions according to the preset historical preference weight and interface state correlation weight to generate a sorted instruction priority list; If the probability value of the instruction with the highest priority in the instruction priority list exceeds the preset probability value, use it as the user's true operation intention; Generate a corresponding control instruction based on the determined user's true operation intention.

[0057] In one implementation, perform a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the performed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the photo frame function operation is performed, cancel the performed photo frame function operation and mark the corresponding control instruction as an error sample, including: Obtain the control instruction and call the photo frame function module to perform the corresponding operation and update the interface state; synchronously record the operation log, and the operation log includes user ID, instruction content, execution status, and timestamp information; After executing the control instruction, start the eye movement tracking device to collect the sequence of the user's fixation point coordinates in real time; extract the eye movement data within a preset time period after the operation for analysis; Perform clustering analysis on the collected eye movement data, and use the K-means algorithm to divide the fixation areas; calculate the Euclidean distance between the current fixation center and the preset cancellation area in the interface; if the fixation center is within the cancellation area and the fixation duration exceeds the preset duration, it is determined as a cancellation-type eye movement behavior; If a cancellation-type eye movement behavior is recognized, it is determined that the execution of this control instruction does not meet the user's expectation; cancel the executed photo frame function operation according to the instruction identifier in the operation log; Mark this control instruction and its associated eye movement behavior data as error samples and store them in the error sample database.

[0058] In one implementation, the method further includes: Obtain the user's recent several operation sequences; use the recent several operation sequences as context inputs; Initialize the long short-term memory neural network model; Input the recent several operation sequences into the LSTM model; the LSTM model learns and extracts the temporal dependence relationship between operations, and outputs a vector representing the context features of the current operation sequence; the vector represents the historical pattern of the user's operations and its temporal correlation.

[0059] Obtain the state information of the current interface in real time; Concatenate the operation context feature vector output by the LSTM model with the current interface state information to form a new comprehensive feature vector; Input the new comprehensive feature vector into the Bayesian probability model to calculate the candidate instruction probability.

[0060] Specifically, obtain the user's recent several operation sequences, use the operation sequences as context inputs, and input them into a pre-initialized long short-term memory neural network (LSTM) model. Among them, the context input refers to the historical data used to model the temporal continuity of the user's operation behavior; the LSTM model is a deep learning model specifically for processing temporal data, with the ability to remember long-term dependence relationships, and can automatically extract the temporal features and potential rules between the user's operations.

[0061] Through the training and inference processes, the LSTM model can learn and extract the temporal dependence relationship in the operation sequence, and output a feature vector representing the context features of the current operation. The feature vector is a high-dimensional numerical vector, which characterizes the historical pattern of the user's recent operations and its correlation in the time dimension, such as whether there is a tendency to continuously execute the same type of operations, or whether there is a preference for a certain specific operation order.

[0062] Collect the status information of the current interface. The status information of the interface includes but is not limited to: the function module where the device is currently located, the position layout of each function area on the interface, whether there is an unexecuted operation queue, and the currently displayed content.

[0063] To realize the joint modeling of context information and interface status, the system concatenates the operation context feature vector output by the LSTM model with the current interface status information to form a new comprehensive feature vector.

[0064] Input the comprehensive feature vector into the Bayesian probability model to calculate the probability of candidate instructions.

[0065] Through the above method, the system can not only identify the possible instructions corresponding to the current eye movement behavior, but also effectively improve the recognition accuracy of ambiguous or multi - meaning eye movement behaviors by combining the user's recent operation habits and interface status changes.

[0066] Refer to Figure 2 , the second embodiment of the present invention provides an intelligent interactive control system for an electronic photo frame based on eye movement tracking, including: The first processing module is used to: track the position of the eyeball in real - time based on the user's eye image data, and obtain the eye movement trajectory sequence according to the change of the pupil center coordinates and the change of the eyelid opening and closing state; The second processing module is used to: calculate the fixation point coordinates, fixation duration and saccade frequency based on the eye movement trajectory sequence; obtain the current eye movement behavior type of the user through a personalized recognition model; match the current eye movement behavior type of the user with a preset eye movement pattern library to obtain a set of candidate instructions; The third processing module is used to: if there are multiple candidate instructions in the set of candidate instructions, analyze the user's historical operation preferences and the current interface status information to rank the priorities of each candidate instruction; determine the candidate instruction with the highest priority as the user's true operation intention and generate a corresponding control instruction; The fourth processing module is used to: execute the photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to judge whether the result of the executed photo frame function operation meets the expectation. If a cancellation - type eye movement behavior is recognized after the execution of the photo frame function operation, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

[0067] It should be noted that an intelligent interactive control system for an electronic photo frame based on eye movement tracking provided by an embodiment of the present invention is used to execute all the process steps of an intelligent interactive control method for an electronic photo frame based on eye movement tracking in the above - mentioned embodiment. Their working principles and beneficial effects correspond one by one, so they will not be elaborated here.

[0068] The specific embodiments described above further elaborate on the object, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only for the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. In particular, it is pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent interactive control method for an electronic photo frame based on eye movement tracking, characterized in that, Including: Real-time tracking of the eye position based on the user's eye image data, and obtaining an eye movement trajectory sequence according to the change of the pupil center coordinates and the change of the eyelid opening and closing state; Calculating the fixation point coordinates, fixation duration, and saccade frequency based on the eye movement trajectory sequence; And obtaining the current eye movement behavior type of the user through a personalized recognition model; Matching the current eye movement behavior type of the user with a preset eye movement pattern library to obtain a set of candidate instructions; If there are multiple candidate instructions in the set of candidate instructions, analyze the user's historical operation preferences and the current interface state information to sort the priorities of each candidate instruction; Determine the real operation intention of the user by taking the candidate instruction with the highest priority and generate a corresponding control instruction; Execute the photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the execution of the photo frame function operation, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

2. The intelligent interaction control method of an electronic photo frame based on eye movement tracking according to claim 1, characterized in that, The construction steps of the personalized recognition model include: Obtaining the user's error samples and correct operation samples to form a data set for constructing a personalized eye movement model; the data set includes several eye movement features, and the eye movement features include fixation point coordinates, fixation duration, and saccade frequency; Extracting the eye movement features from the error samples and correct operation samples and constructing an initial eye movement model; Analyze the eye movement features, calculate the weight coefficients of each eye movement feature to obtain a feature weight distribution; Judge whether the feature weight distribution deviates from a preset threshold. If it deviates, adjust the weight coefficient according to the user's operation habit to generate an optimized eye movement model; Based on the optimized eye movement model, adjust the instruction recognition threshold in combination with the user's operation habit to obtain a personalized recognition model; Process the real-time eye movement data through the personalized recognition model, judge whether the instruction recognition result conforms to the user's operation habit to generate a matching result; update the eye movement feature weights and instruction recognition thresholds of the personalized recognition model according to the matching result; If the matching result of the personalized recognition model does not reach the preset accuracy rate, repeat the above steps to optimize the model parameters until the final personalized recognition model is obtained.

3. The intelligent interaction control method of an electronic photo frame based on eye movement tracking according to claim 2, characterized in that The method further includes: Real-time collecting the user's interaction data through the system background; extracting the user identification information and historical recognition accuracy rate based on the user's interaction data; if the user's recognition accuracy rate is lower than the preset threshold or the user is a newly registered user, trigger the progressive learning mode; In the progressive learning mode, according to the user's current operation level, screen out the core function instructions from the preset instruction library to generate a simplified instruction sequence; generate corresponding visual guidance elements for each simplified instruction; adjust the form and intensity of the visual feedback based on the user's eye movement features to match the user's operation habit; the simplified instruction sequence covers the most basic and commonly used interaction operations. During the process of the user using simplified instructions, eye movement trajectories and operation data are collected in real time, and the user proficiency index is calculated; when the user's proficiency reaches the proficiency threshold, it is determined that the current instruction set has been mastered; After it is determined that the current instruction set has been mastered, the system will gradually unlock more complex instructions from the preset instruction library and update the visual feedback interface to support the learning of new instructions.

4. The intelligent interaction control method of an electronic photo frame based on eye movement tracking according to claim 3, characterized in that, The method further includes: Analyze the scenarios where the user's instruction execution fails to identify the error type and its cause; Adjust the parameters of the personalized recognition model based on the error type distribution and adjust the instruction mapping rules in combination with the user's operation habits.

5. The intelligent interactive control method of an electronic photo frame based on eye movement tracking according to claim 4, wherein, Based on the user's eye image data, the position of the eyeball is tracked in real time, and an eye movement trajectory sequence is obtained according to the change of the pupil center coordinates and the change of the eyelid opening and closing state, including: Collect the user's eye image data through the camera module; Perform image preprocessing on the collected image to obtain the first image; Based on the YOLOv5 model of deep learning, detect the pupil area of the first image to obtain the pupil bounding box coordinates and calculate the pupil center coordinates according to the center of the bounding box; Detect the distance between the upper and lower eyelids to judge the eyelid opening and closing state; Generate a timestamped eye movement trajectory sequence according to the displacement change of the pupil center coordinates and the time series of the eyelid opening and closing state in consecutive frames; where, when the eye movement trajectory sequence is interrupted, trigger a repositioning mechanism to rescan the first image until stable pupil coordinates are obtained; Use the Kalman filter algorithm to smooth the trajectory sequence to obtain the second trajectory sequence; Detect the Euclidean distance of the pupil displacement between adjacent frames of the second trajectory sequence to identify abnormal points, and use the interpolation method to repair the trajectory breakpoints to form a continuous and stable third trajectory sequence; Extract eye movement feature parameters according to the change rate of the pupil center coordinates and the eyelid opening and closing frequency of the third trajectory sequence.

6. The intelligent interactive control method of an electronic photo frame based on eye movement tracking according to claim 5, wherein, Calculate the fixation point coordinates, fixation duration, and saccade frequency based on the eye movement trajectory sequence; And obtain the user's current eye movement behavior type through the personalized recognition model; Match the user's current eye movement behavior type with the preset eye movement pattern library and obtain a candidate instruction set, including: Calculate the fixation time, saccade speed, and detect the blink frequency according to the eye movement trajectory sequence: Based on the personalized recognition model, compare the calculated feature vector with the preset eye movement pattern library, and use the K-nearest neighbor algorithm to calculate the similarity between the current feature vector and each pattern in the pattern library; If the similarity of the features in a certain period to a certain pattern in the pattern library exceeds the preset threshold, it is determined as the corresponding eye movement behavior type; For the identified eye movement behavior type, screen out a set of candidate instructions associated with it from the preset instruction library through the mapping relationship.

7. An intelligent interaction control method for an electronic photo frame based on eye movement tracking according to claim 6, characterized in that, If there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and the current interface state information to sort the priorities of each candidate instruction; Determine the candidate instruction with the highest priority as the user's true operation intention and generate the corresponding control instruction, including: Obtain the user's historical operation data and the current interface state information; Perform standardization processing on the collected operation data and interface state information to form an input data set; Calculate the initial probability distribution of each candidate instruction based on the standardized input data set using a Bayesian probability model to obtain the probability values of each instruction; Use the weighted summation method to prioritize the candidate instructions according to the preset historical preference weights and interface state correlation weights to generate a sorted instruction priority list; If the probability value of the instruction with the highest priority in the instruction priority list exceeds the preset probability value, then use it as the user's true operation intention; Generate a corresponding control instruction based on the determined user true operation intention.

8. The intelligent interactive control method of an electronic photo frame based on eye movement tracking according to claim 7, wherein, Execute the photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets the expectation. If a cancellation-type eye movement behavior is recognized after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample, including: Obtain the control instruction and call the photo frame function module to execute the corresponding operation and update the interface state; synchronously record the operation log, and the operation log includes user ID, instruction content, execution status, and timestamp information; After executing the control instruction, start the eye movement tracking device to collect the sequence of the user's fixation point coordinates in real time; extract the eye movement data within a preset time period after the operation for analysis; Perform clustering analysis on the collected eye movement data, and use the K-means algorithm to divide the fixation area; calculate the Euclidean distance between the current fixation center and the preset cancellation area in the interface; if the fixation center is within the cancellation area and the fixation duration exceeds the preset duration, then determine it as a cancellation-type eye movement behavior; If a cancellation-type eye movement behavior is recognized, then determine that the execution of this control instruction does not meet the user's expectation; cancel the executed photo frame function operation according to the instruction identifier in the operation log; Mark this control instruction and its associated eye movement behavior data as error samples and store them in the error sample database.

9. The intelligent interactive control method of an electronic photo frame based on eye movement tracking according to claim 8, characterized in that, The method further includes: Obtain the user's recent several operation sequences; use the recent several operation sequences as context input; Initialize the long short-term memory neural network model; Input the recent several operation sequences into the LSTM model; the LSTM model learns and extracts the temporal dependency relationship between the operations and outputs a vector representing the context features of the current operation sequence; the vector represents the historical pattern of the user's operations and their temporal correlation; Obtain the state information of the current interface in real time; Concatenate the operation context feature vector output by the LSTM model with the current interface state information to form a new comprehensive feature vector; Input the new comprehensive feature vector into the Bayesian probability model to calculate the candidate instruction probability.

10. An intelligent interactive control system for an electronic photo frame based on eye movement tracking, which is used to execute the method according to any one of claims 1-9, characterized in that, Including: The first processing module is used to: perform real-time tracking on the position of the eyeball based on the user's eye image data, and obtain the eye movement trajectory sequence according to the change of the pupil center coordinates and the change of the eyelid opening and closing state; The second processing module is used to: calculate the fixation point coordinates, fixation duration, and saccade frequency based on the eye movement trajectory sequence; And obtain the user's current eye movement behavior type through a personalized recognition model; Match the user's current eye movement behavior type with the preset eye movement pattern library to obtain a candidate instruction set; A third processing module, configured to: if there are multiple candidate instructions in the candidate instruction set, analyze the user's historical operation preferences and the current interface state information to perform a priority ranking on each candidate instruction; determine the candidate instruction ranked first in the priority ranking as the user's true operation intention and generate a corresponding control instruction; A fourth processing module, configured to: perform a photo frame function operation according to the control instruction; monitor the user's subsequent eye movement behavior to determine whether the result of the executed photo frame function operation meets the expectation, and if a cancellation-type eye movement behavior is recognized after the photo frame function operation is executed, cancel the executed photo frame function operation and mark the corresponding control instruction as an error sample; the error sample is used to update the model parameters of the personalized recognition model.

Images