Eye-tracking-based eye-controlled interaction method, device, and system

Abstract

The present invention provides an eye control interaction method, device and system based on eye movement tracking, which relates to the field of data processing technology. The method includes: collecting eye movement data and user feature data of the subject; the user feature data includes age, pupil color and type of eye disease; extracting eye movement feature data based on the eye movement data and user feature data; generating brain activity state information of the subject based on the eye movement feature data, determining the eye control instructions of the subject based on the brain activity state information, eye movement feature data and user feature data; and performing eye control interaction based on the eye control instructions. The present invention can take into account the characteristics of brain activity state and eye diseases that may exist in different types of users, collect user feature data and perform feature extraction in combination with eye movement data, obtain eye movement feature data that accurately reflects the user's true eye movement intention, and adapt the obtained eye control instructions to different users, thereby improving the accuracy of eye control interaction and bringing users a smoother and more natural interaction experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an eye-tracking-based eye-controlled interaction method, device, and system. Background Art

[0002] Eye tracking technology is rapidly developing in fields such as human-computer interaction and medical-industrial integration, becoming a key tool for enhancing the naturalness and immersion of interactions. In recent years, with the rise of virtual reality (VR) and augmented reality (AR) technologies, the application scenarios of eye tracking technology have continued to expand. For example, eye tracking technology is increasingly being used in game design, not only enhancing the immersive and interactive experience, but also helping developers optimize game design and enhance the user experience. Furthermore, eye tracking technology is increasingly being used in neurology (neurology) and other medical fields, demonstrating its significant value in the early assessment of neurodegenerative diseases and cognitive function.

[0003] In related technologies, the method of realizing human-computer interaction based on eye tracking technology is usually: the human-computer interaction device obtains human eye movement data, extracts eye movement features from the eye movement data, generates corresponding eye control instructions by identifying eye movement features, and realizes human-computer interaction through eye control instructions.

[0004] While eye tracking technology has shown great potential in areas such as human-computer interaction and medical-industrial integration, it still faces numerous challenges. These challenges are primarily due to differences between users: First, eye movement data is regulated by the brain, and some users may experience abnormal brain activity (such as abnormalities in attention allocation and inhibitory control), resulting in normal limb movement behavior but subtle anomalies in eye movement data. Second, some users may suffer from eye diseases, which can also cause abnormal eye movement data. These differences can cause the eye movement features extracted by eye tracking technology, and the eye control commands it identifies and generates, to differ from the user's actual intentions. Currently, the calibration function of eye tracking devices makes it difficult to accurately adjust relevant control commands for different users, resulting in low recognition accuracy and difficulty adapting to diverse user and scenario requirements. Summary of the Invention

[0005] The embodiments of the present invention provide an eye-controlled interaction method, device, and system based on eye tracking to solve the problem of low recognition accuracy of eye tracking during the interaction process at this stage.

[0006] In a first aspect, an embodiment of the present invention provides an eye-controlled interaction method based on eye tracking, comprising:

[0007] Collecting the subjects' eye movement data and user characteristic data; the user characteristic data includes age, pupil color, and type of eye disease;

[0008] Extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data;

[0009] generating brain activity state information of the subject based on the eye movement feature data; and determining eye control instructions of the subject based on the brain activity state information, the eye movement feature data, and the user feature data;

[0010] Perform eye-controlled interaction based on eye-controlled commands.

[0011] In a possible implementation, extracting eye movement feature data from the eye movement data based on the eye movement data and the user feature data includes:

[0012] The pupil light reflex was determined based on the subject's pupil color;

[0013] Enhance the pupil boundary in the eye movement data based on the pupil light reflex;

[0014] Determine the extraction index according to the age of the subject; and extract features from the eye movement data based on the extraction index to obtain initial eye movement feature data;

[0015] Determine impact data based on the type of eye disease;

[0016] Eye movement feature data is obtained based on the impact data and the initial eye movement feature data.

[0017] In one possible implementation, the extraction indicators are determined based on the age of the subject, including:

[0018] If the subject's age is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators;

[0019] If the subject's age is within the second preset age range, saccade amplitude, frequency, fixation time, scanning speed, and visual search efficiency are used as extraction indicators;

[0020] If the subject's age is within the third preset age range, blink frequency, adaptation time when switching gaze to targets at different distances, gaze point stability, saccade speed, and gaze time are used as extraction indicators.

[0021] In a possible implementation, the impact data is determined based on the type of eye disease, including:

[0022] If the eye disease type is myopia, the affected data are the amplitude and frequency of saccades and the adaptation time when switching gaze to targets at different distances;

[0023] If the eye disease type is hyperopia, the affected data are saccade amplitude, frequency, and fixation time;

[0024] If the eye disease type is astigmatism, the affected data are saccade amplitude, frequency, and fixation stability;

[0025] If the eye disease type is glaucoma, the affected data are saccade amplitude, frequency, saccade speed, and the area of fixation;

[0026] If the eye disease type is cataract, the affected data are eye movement speed, gaze stability, and blink frequency.

[0027] In one possible implementation, obtaining eye movement feature data based on the impact data and the initial eye movement feature data includes:

[0028] If the eye disease type is myopia, then based on the subject's historical eye movement data and the predicted value of the eye movement data at the next moment, the abnormal eye movement data in the initial eye movement feature data is removed. At the same time, according to the degree of myopia, the adaptation time when switching gaze to targets at different distances is adjusted to obtain the eye movement feature data;

[0029] If the eye disease type is hyperopia, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced based on the subject's average mixed eye movement data. At the same time, the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data.

[0030] If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, the abnormal fixation point data in the initial eye movement feature data is removed, and the focus of the saccade is corrected based on the degree of astigmatism to obtain the eye movement feature data;

[0031] If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the eye movement trajectory is corrected based on the relationship between the eye movement data and the missing fixation point area. At the same time, the scanning speed is adjusted based on the severity of the disease to obtain eye movement feature data;

[0032] If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the rotation frequency data is adjusted based on the average blink frequency of the subject to obtain the eye movement feature data.

[0033] In one possible implementation, determining the subject's eye control command based on the brain activity state information, the eye movement feature data, and the user feature data includes:

[0034] If the brain activity state information is normal, determining the correlation priority between each feature data in the user feature data and the eye control command;

[0035] The subject's eye control instructions are determined based on the relevance priority of each feature data and the eye movement feature data.

[0036] In one possible implementation, determining the subject's eye control instructions based on the relevance priorities of various feature data and the eye movement feature data includes:

[0037] Match each feature data and eye movement feature data with the rules in the preset rule library to obtain the target eye control instruction;

[0038] The subject's eye control instructions are determined based on the relevance priority of each feature data.

[0039] In one possible implementation, determining the relevance priority between each feature data in the user feature data and the eye control command includes:

[0040] Analyze the correlation between various features in the user feature data and eye control commands;

[0041] According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0042] In a second aspect, an embodiment of the present invention provides an eye-controlled interaction device based on eye tracking, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method in the first aspect or any possible implementation of the first aspect.

[0043] In a third aspect, an embodiment of the present invention provides an eye-tracking-based eye-controlled interaction system, an eye tracker, and an eye-tracking-based eye-controlled interaction device as provided in the second aspect above.

[0044] In an embodiment of the present invention, the characteristics of different types of users are taken into account, and multiple user feature data is collected and combined with eye movement data for feature extraction. This results in eye movement feature data that accurately reflects the user's true eye movement intentions. This data can avoid command misjudgments caused by ignoring individual differences, significantly improving the accuracy of determining eye control commands and making interactions more accurate. When determining eye control commands, brain activity status information is generated based on the eye movement feature data to determine the user's brain health status (for example, whether there are abnormalities such as inattention, frequent switching of gaze points, inability to accurately focus on the target area, and slow reaction speed). Eye control commands are then generated based on the brain activity status information, user feature data, and eye movement feature data. This again takes into account the characteristics of different users so that the resulting eye control commands are adapted to different users, improving the accuracy of eye control interactions while providing users with a smoother and more natural interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 1 is a system architecture diagram of an eye-tracking-based eye-controlled interaction system provided by an embodiment of the present invention;

[0046] Figure 2 is a flowchart of an implementation of an eye-controlled interaction method based on eye tracking provided by an embodiment of the present invention;

[0047] Figure 3 This is a flowchart for extracting eye movement feature data according to an embodiment of the present invention;

[0048] Figure 4 1 is a schematic structural diagram of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of the structure of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] In the related art, when implementing human-computer interaction based on eye tracking technology, the same eye movement features are typically extracted for different users. Eye control commands are then generated based on these extracted features, and human-computer interaction is achieved according to these eye control commands. However, different users may differ, primarily due to physiological and pathological factors. Pathological factors primarily include abnormal brain activity and eye diseases. Eye movement data is regulated by the brain, and some users may have abnormal brain activity (such as inattention, frequent gaze switching, inability to accurately focus on the target area, and slow reaction times). This may result in normal limb movement behavior but subtle anomalies in eye movement data. Furthermore, some users may have eye diseases (such as myopia and glaucoma) that can also cause abnormal eye movement data. Physiological factors primarily include pupil characteristics and age. These differences can cause the eye movement features extracted and the eye control commands generated by eye tracking technology to mismatch the user's actual intentions. Currently, the calibration function of eye tracking devices struggles to accurately adjust the relevant control commands for different users, resulting in low recognition accuracy and difficulty adapting to diverse user and scenario needs.

[0052] To address the above-mentioned shortcomings, this application proposes the following technical concepts: When extracting eye movement feature data, the characteristics of different types of users are taken into consideration. Multiple user feature data (including age, pupil color, and type of eye disease) is collected and combined with the eye movement data for feature extraction. This results in eye movement feature data that accurately reflects the user's true eye movement intentions. This data can avoid command misjudgments caused by ignoring individual differences, significantly improving the accuracy of determining eye control commands and making interactions more accurate. When determining eye control commands, brain activity state information is generated based on the eye movement feature data to determine the user's brain health status (for example, whether there are abnormalities such as inattention, frequent switching of gaze points, inability to accurately focus on the target area, and slow reaction speed). Eye control commands are then generated based on the brain activity state information, user feature data, and eye movement feature data. This again takes into account the characteristics of different users so that the resulting eye control commands are adapted to different users, improving the accuracy of eye control interactions while providing users with a smoother and more natural interactive experience.

[0053] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a system architecture diagram of an eye-tracking-based eye-controlled interaction system provided by an embodiment of the present invention. Figure 1 As shown, the system may include an eye tracker 101 and an eye-tracking-based eye-control interaction device 102. During application, the eye tracker 101 may collect the subject's eye movement data, which is then transmitted to the eye-tracking-based eye-control interaction device 102. The eye-tracking-based eye-control interaction device 102 then analyzes the eye movement data to obtain eye control commands. The eye tracker will display the corresponding screen. When the eye control command is being controlled, the eye tracker will be able to display the control process, allowing the subject to determine whether the recognized eye control command is correct. If it is incorrect, adjustments can be made in a timely manner.

[0055] In this embodiment, the eye tracker can be in any of three forms: open, desktop, or stand. These three forms share the same components, operating principles, and mechanisms, differing only in their integration methods. Each eye tracker can include a host, acquisition devices, and display devices. In actual use, different eye trackers can be deployed and installed based on user needs.

[0056] The following examples illustrate the installation process of different eye trackers:

[0057] For example, the installation process of an open-type eye tracker is as follows: prepare a power strip with more than four ports, a table for placing the subject's monitor, etc., and a desktop for placing the monitor, etc.; install the subject's monitor, monitor, computer, camera and light source, and head and jaw support; connect the data transmission according to the instructions; connect the power cords of all devices that need power to the power strip; conduct an acceptance test, boot into the system, and start the eye tracker software to check whether the video image and various functions are normal.

[0058] The installation process for a desktop eye tracker is as follows: prepare a power strip with at least four ports, a tabletop for the desktop device, and a desktop for the monitor. Install the desktop device, monitor, and computer. Connect the camera and data cables according to the instructions. Connect the power cords of all devices requiring power to the power strip. Perform an acceptance test, boot the system, and launch the eye tracker software to check whether the video image and various functions are functioning properly.

[0059] The installation process for a standing eye tracker is as follows: prepare a power strip with at least four ports, a desk for a monitor, and space for the standing device; install the standing device, such as disengaging the universal wheel brakes and the monitor; wire the equipment and connect the data transmission cable according to the instructions; connect the power cords of all devices requiring power to the power strip; perform an acceptance test, boot the system, and then launch the eye tracker software to check whether the video image and various functions are functioning normally.

[0060] In this embodiment, the eye-tracking-based eye-controlled interaction device 102 includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements an eye-movement data analysis method to obtain eye-control instructions for eye-controlled interaction.

[0061] Alternatively, an embodiment of the present invention can also be implemented based on a near-eye display device, which includes an eye-tracking-based eye-controlled interaction device 102. The eye-tracking-based eye-controlled interaction device 102 collects eye movement data through the near-eye display device, and obtains eye control instructions based on the collected near-eye display device and user feature data, so that the near-eye display device can implement eye control interaction based on the eye control instructions.

[0062] Figure 2 This is a flow chart of the implementation of the eye-tracking-based eye-control interaction method provided by an embodiment of the present invention. Figure 2 The method implemented by the eye-tracking-based eye-controlled interaction device 102 is described below:

[0063] Step S201: collecting the subject's eye movement data and user characteristic data; wherein the user characteristic data includes age, pupil color, and eye disease type.

[0064] In this embodiment, the subject's eye movement data can be collected through an eye tracker or a near-eye display device.

[0065] For example, eye movement data can be obtained through the eye tracker's specific interface functions and presented in an array format. Each row represents different eye movement information, including the frame number, the X and Y coordinates of the left and right pupils, pupil diameter and area, the X and Y coordinates of the left and right eye reflection points, and the X and Y coordinates of the left and right eye viewing angles. These parameters accurately record key information such as eye position, pupil size, and viewing angle direction in each frame, providing a detailed data foundation for subsequent eye movement analysis.

[0066] Alternatively, eye movement data can be stored in the form of videos and images through an eye tracker. Video files intuitively display the state of the eyes at different moments. By analyzing videos and images, we can observe information such as eye movement trajectory and pupil changes. These complement the array data and provide researchers with a more comprehensive perspective on eye movement data.

[0067] In this embodiment, the user characteristic data may be information entered by the subject during use, which may include the user's age, pupil color, and type of eye disease; wherein, eye diseases may include normal, myopia, hyperopia, astigmatism, glaucoma, cataracts, etc.

[0068] Step S202: extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data.

[0069] In this embodiment, the traditional method extracts common eye movement features for different users. However, due to differences in age, pupil color, and eye diseases among different users, using the same features is not sufficient to accurately represent the intentions of each user. Therefore, the method in this embodiment fully considers the differences among different users in terms of age, pupil color, and eye disease types. These user feature data provide an important basis for in-depth analysis of eye movement data.

[0070] Eye movement characteristics vary across age groups. Younger people may have faster eye movement reactions, while older people may have slower eye movements and less stable gaze. By incorporating age into the analysis, we can more accurately extract eye movement characteristics for users of different age groups.

[0071] Different pupil colors reflect light differently, which can affect the collection and analysis of eye movement data. For example, lighter pupils may reflect more light under the same lighting conditions, resulting in different characteristics in the collected eye movement data.

[0072] Eye disease types directly influence eye movement patterns. Conditions like myopia, hyperopia, astigmatism, glaucoma, and cataracts can cause eye movements to exhibit distinct characteristics in terms of amplitude, frequency, and fixation duration. Extracting eye movement features from these user-specific data comprehensively reflects the eye movement characteristics of different users, avoiding analysis errors caused by ignoring individual differences.

[0073] Step S203: Generate the subject's brain activity state information based on the eye movement feature data.

[0074] In this step, the eye movement feature data may include, but is not limited to, saccade amplitude, saccade frequency, adaptation time when switching gazes between targets at different distances, gaze duration, gaze point stability, scan rate, gaze point location, and blink frequency.

[0075] Because eye movements are coordinated and controlled by multiple regions of the cerebrum and cerebellum (including the frontal eye movement areas, parietal lobes, basal ganglia, thalamus, and brainstem), these eye movement data can reflect specific brain functions, such as attention allocation, visual information processing, emotional state, memory strength, inhibitory control, and other cognitive and physiological conditions. Therefore, these eye movement data can reveal whether a subject's brain activity (i.e., nervous system function) is abnormal. For example, Parkinson's disease, due to dopamine deficiency affecting eye movement control (resulting in abnormalities in brain activity such as inhibitory control), may be characterized in its early stages by small increases in saccade amplitude and frequency, slower saccade rate, and altered blink rate. Alzheimer's disease, due to impaired attention and visual processing (resulting in abnormalities in brain activity such as attention allocation), may be characterized in its early stages by prolonged adaptation time when switching between different targets, prolonged fixation duration, and a dispersed focus area.

[0076] In a possible implementation, a judgment threshold range is set for each eye movement feature, the eye movement feature is compared with the corresponding judgment threshold range, and brain activity state information is generated according to the comparison result.

[0077] For example, the range of normal eye movement characteristics is usually: eye movement amplitude range is 5°~30°, eye movement frequency is 1~2 times / second in the natural state, scanning rate is 200°~400° / second, adaptation time when switching gaze between targets at different distances is usually less than 500 milliseconds, gaze time is: 200~500ms / time in the target area, gaze point stability is: jitter amplitude <1° (visual angle), the area where the gaze point is located is concentrated in the task-related area of interest, and the blinking frequency is 15~20 times / minute. If all eye movement feature data are within the corresponding judgment threshold range, the generated brain activity status information is: normal; if the fixation time of 30% of the target area is less than 100 milliseconds, the area where the fixation point is located deviates from the target area (the hit rate of the area of interest is less than 60%), the adaptation time when switching between different targets is greater than 1.2 seconds, and the eye saccade frequency is greater than 3 times / second, the generated brain activity status information is: the first abnormal situation; if the peak rate in the scanning rate is less than 180° / second, the eye saccade amplitude in 40% of the eye saccades is less than 5°, the stability of the fixation point decreases (the jitter amplitude is greater than 1.5°), and the blinking frequency decreases (less than 12 times / minute), the generated brain activity status information is: the second abnormal situation.

[0078] Step S204: determining the subject's eye control instructions based on the brain activity state information, the eye movement feature data, and the user feature data.

[0079] In this embodiment, because the association between eye movement characteristics and true intentions varies among users of different ages, pupil colors, types of eye diseases, and different brain activity states, determining eye control commands based solely on general eye movement characteristics may lead to inaccurate command interpretation and fail to meet the actual needs of diverse users. Therefore, this embodiment also takes user characteristic data into consideration when determining eye control commands.

[0080] Step S205: performing eye control interaction based on the eye control instruction.

[0081] After determining the eye control command, corresponding interactive operations can be performed according to different application scenarios. For example, in a game scene, if the eye control command is to aim at a target, the system will accurately position the game character's perspective or crosshairs to the corresponding target based on the gaze point position determined by the eye movement data, thereby achieving rapid aiming. In a VR or AR scene, if the eye control command is to select an object in the virtual scene, the system will determine the object the user is looking at based on the eye movement data, and then trigger the corresponding selection operation, which may be accompanied by feedback such as highlighting the object and displaying information, so that the user clearly knows that the operation has been executed. In some smart device control scenarios, if the eye control command is to open an application, the system will recognize the eye control command and then start the corresponding application.

[0082] Figure 3is a flowchart of eye movement feature data extraction provided by an embodiment of the present invention; Figure 3 As shown, in an optional embodiment, extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data in step S202 may include:

[0083] Step S2021: Determine pupil light reflexivity based on the subject's pupil color.

[0084] Step S2022: enhancing the pupil boundary in the eye movement data according to the pupil light reflexivity.

[0085] Step S2023: determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data.

[0086] Step S2024: Determine impact data based on the type of eye disease.

[0087] Step S2025: Obtain eye movement feature data based on the impact data and the initial eye movement feature data.

[0088] In this embodiment, changes in pupil size are an indicator of the brain's response to environmental changes and are closely linked to arousal, emotion, and cognitive function. When extracting eye movement features, accurate pupil center detection combined with other feature points, such as corneal reflection points, allows for more accurate calculation of other eye movement features, such as gaze direction and fixation point.

[0089] However, different subjects have different pupil colors, resulting in different sensitivities to light. Moreover, in eye movement images, the pupil boundaries of light-colored pupils are not obvious in strong light, while the pupil boundaries of dark-colored pupils are not obvious in dark light, which leads to inaccurate recognition during extraction.

[0090] Therefore, considering the inherent connection between pupil color and pupil light reflectivity, generally speaking, lighter pupils, such as blue pupils, have relatively higher light reflectivity, while darker pupils, such as black pupils, have relatively lower light reflectivity. Based on this correlation, combined with existing research data or pre-established models, it is possible to determine the light reflectivity of the subject's pupil. This can provide a more accurate basis for the subsequent extraction of eye movement feature data based on the enhanced boundaries. For example, when analyzing the amplitude and frequency of eye saccades, changes in eye position can be more accurately determined. In eye control operations on smart devices, the user's gaze point can be more accurately identified, achieving precise control.

[0091] In this embodiment, taking into account the different ways in which different age groups understand and express intentions, the eye movement data expressed by subjects of different age groups have different characteristics. In order to ensure that the extracted features can accurately reflect the intentions of the subjects, this embodiment designs different feature extraction indicators for subjects in different age groups.

[0092] Based on the age of the subjects, extraction indicators are determined, and features are extracted from the eye movement data based on the extraction indicators to obtain initial eye movement feature data.

[0093] Then, considering that eye diseases will affect eye movement data, the intention actually expressed by the subject will deviate from the intention detected under the influence of eye diseases. Therefore, this embodiment determines the impact of each type of eye disease on eye movement characteristics according to different types of eye diseases, that is, the impact data, and corrects the initial eye movement feature data based on the impact data to obtain the eye movement feature data.

[0094] In an optional embodiment, determining the extraction index according to the age of the subject in step S2023 may include:

[0095] If the subject's age is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators.

[0096] If the subject's age is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators.

[0097] If the subject's age is within the third preset age range, blink frequency, adaptation time when switching gaze to targets at different distances, gaze point stability, saccade speed, and gaze time are used as extraction indicators.

[0098] In this embodiment, three age intervals are set based on the characteristics of age groups. The first preset age interval represents the childhood stage, the second preset age interval represents the adolescent stage, and the third preset age interval represents the adult stage. The first preset age interval can be 0-12 years old, the second preset age interval can be 13-18 years old, and the third preset age interval can be 19 years old and above. Of course, in actual application, these can also be adjusted according to actual needs.

[0099] During the childhood stage, since children understand and express their intentions differently from adults, their eye movements may be more random. When determining eye control instructions, considering their easily distracted and frequent eye saccades, multiple short-term gazes at the same area can be determined as an operational intention, and the instruction setting is relatively simple and direct. For example, in educational games, multiple gazes at a certain knowledge point icon in a short period of time can be determined as an eye control instruction to obtain a detailed explanation of the knowledge point. Therefore, for subjects in this age group, the amplitude and frequency of eye saccades can be used as extraction indicators. Accordingly, the initial eye movement feature data can be data that can reflect the amplitude and frequency of eye saccades.

[0100] Adolescents have active minds and a near-maturity visual system, but they also employ unique strategies for complex tasks. Indicators such as the time to first fixation on the target, the length of the search path, and the number of repeated fixations can be extracted. Furthermore, given the changes in adolescents' concentration during long-duration tasks, additional analysis of the distribution of fixation time can be performed, such as by statistically analyzing changes in fixation time during different task phases. Therefore, for subjects in this age group, saccade amplitude, frequency, fixation duration, scan speed, and visual search efficiency can be used as extraction indicators. Accordingly, the initial eye movement feature data can be data that reflects saccade amplitude, frequency, fixation duration, scan speed, and visual search efficiency.

[0101] Adults will experience fatigue during long-term visual tasks, so changes in gaze stability can be extracted, such as the frequency of small movements of the gaze point, the increase in blinking frequency, smoothness indicators of the start and end of saccades, such as changes in acceleration during saccades, and visual search efficiency in complex scenes. Therefore, for subjects in this age group, blinking frequency, adaptation time when switching gazes between targets at different distances, gaze point stability, saccade speed, and gaze time can be used as extraction indicators. Accordingly, the initial eye movement feature data can be data that can reflect blinking frequency, adaptation time when switching gazes between targets at different distances, gaze point stability, saccade speed, and gaze time.

[0102] In an optional embodiment, determining the impact data according to the eye disease type in step S2024 may include:

[0103] If the eye disease type is myopia, the affected data are the amplitude and frequency of saccades and the adaptation time when switching gaze to targets at different distances.

[0104] If the type of eye disease is hyperopia, the affected data are saccade amplitude, frequency, and fixation time.

[0105] If the type of eye disease is astigmatism, the affected data are the amplitude and frequency of saccades and the stability of the gaze point.

[0106] If the type of eye disease is glaucoma, the affected data are saccade amplitude, frequency, scanning speed and the area of gaze.

[0107] If the eye disease type is cataract, the affected data are eye movement speed, gaze stability, and blink frequency.

[0108] In this embodiment, if the subject does not have any eye disease, that is, the eye disease type is healthy, the impact data is empty, or there is no need to determine the impact data, and the eye control command can be determined based on the initial eye movement feature data and the user feature data.

[0109] If the subject is nearsighted, this can alter the eye's refractive power and affect eye movement patterns. Because myopia increases the length of the eye axis, this can affect the amplitude and frequency of saccades. Furthermore, when reading or working at close range, the focus may be closer to the target. Therefore, myopia can affect saccade amplitude and frequency, as well as the adaptation time required to switch between targets at different distances.

[0110] If the subject is hyperopic, the hyperopic eye requires greater accommodation when viewing close objects, which is reflected in eye movements. Hyperopic patients have difficulty accommodating close objects, and are prone to unstable fixation and abnormal saccades. Therefore, hyperopia can affect indicators such as saccade amplitude, frequency, and fixation duration.

[0111] If the subject has astigmatism, the astigmatism will cause the eye's refractive power to vary in different directions, resulting in different eye movement characteristics in different directions. This will lead to errors in the accuracy of saccades and the stability of the gaze point, thus affecting indicators such as saccade amplitude, frequency, and gaze point stability.

[0112] If the subject suffers from glaucoma, it can affect optic nerve function, leading to visual field defects and abnormal eye movements. For such subjects, the area where their gaze is located is imperfect, and this can also affect the amplitude, frequency, and speed of saccades.

[0113] If the subject suffers from cataracts, the cataract will cause the lens to become cloudy, affecting vision, resulting in slower eye movement speed and reduced gaze stability. Therefore, for such subjects, eye movement speed, gaze stability, and blinking frequency can be used as impact data.

[0114] In an optional embodiment, obtaining eye movement feature data based on the impact data and the initial eye movement feature data may include:

[0115] If the eye disease type is myopia, the abnormal saccade data in the initial eye movement feature data is removed based on the subject's historical eye movement data and the predicted value of the eye movement data at the next moment. At the same time, according to the degree of myopia, the adaptation time when switching gaze to targets at different distances is adjusted to obtain eye movement feature data.

[0116] If the eye disease type is hyperopia, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced based on the subject's average mixed eye movement data. At the same time, the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data.

[0117] If the eye disease type is astigmatism, the abnormal gaze point data in the initial eye movement feature data is removed based on the direction and angle of the subject's astigmatism axis, and the eye saccade focus is corrected based on the degree of astigmatism to obtain the eye movement feature data.

[0118] If the eye disease type is glaucoma, the area where the gaze point is missing in the initial eye movement feature data is completed, and the eye movement trajectory is corrected based on the relationship between the eye movement data and the area where the gaze point is missing. At the same time, the scanning speed is adjusted based on the severity of the disease to obtain eye movement feature data.

[0119] If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the rotation frequency data is adjusted based on the average blink frequency of the subject to obtain the eye movement feature data.

[0120] In this embodiment, for subjects with myopia, it is necessary to remove abnormal saccade data from the initial eye movement feature data and correct the adaptation time when switching gazes between targets at different distances. Removing abnormal saccade data from the initial eye movement feature data can be achieved by the following steps:

[0121] Establish a historical eye movement database for each subject, storing data such as saccade amplitude and frequency over a period of time. Use time series analysis or anomaly detection algorithms from machine learning, such as the isolation forest algorithm, to evaluate the saccade data within the initial eye movement feature data and identify outliers that are far away from the majority of data points. Compare the predicted eye movement data for the next moment with the historical data. If the predicted value differs significantly from the historical data and falls outside the abnormal range determined by the isolation forest algorithm, it is considered abnormal and removed.

[0122] Correcting the adaptation time when switching gazes at targets at different distances can be achieved through the following steps: dividing different intervals according to the degree of myopia, such as low myopia is -0.50D to -3.00D, moderate myopia is -3.25D to -6.00D, and high myopia is -6.25D and above. According to clinical research and experimental data, a gaze adaptation time adjustment coefficient is set for each interval. For example, the adjustment coefficient for low myopia is 1.2, for moderate myopia is 1.5, and for high myopia is 1.8. Of course, the above data is only for illustration and can be adjusted as needed in actual applications. After the degree of myopia of the subject is detected, the adaptation time when switching gazes at targets at different distances initially measured is multiplied by the corresponding coefficient to obtain the adjusted adaptation time, thereby completing the optimization of the eye movement feature data.

[0123] For subjects with hyperopia, it is necessary to amplify the saccade amplitude in the initial eye movement feature data and reduce the saccade frequency. At the same time, based on the degree of hyperopia, the fixation time is corrected to obtain the eye movement feature data.

[0124] Among them, adjusting the amplitude and frequency of eye saccades can be achieved by the following steps:

[0125] Calculate the average mixed eye movement data for the subject over a preset time period, including the mean saccade amplitude and frequency. Construct a mapping relationship based on the degree of hyperopia to amplify saccade amplitude and reduce saccade frequency. Adjust the saccade amplitude and frequency in the initial eye movement feature data based on this mapping relationship to make the eye movement characteristics more consistent with the actual hyperopia patient.

[0126] Correction of fixation time can be achieved by following the steps below:

[0127] Experimental studies have found that the higher the degree of hyperopia, the longer the near-distance fixation time should be. For example, a setting can be set to increase the near-distance fixation time by 10% for every 1D increase in hyperopia. After obtaining the subject's hyperopia degree, the near-distance fixation time in the initial eye movement feature data is adjusted accordingly to complete the correction of the eye movement feature data.

[0128] For subjects with astigmatism, it is necessary to remove abnormal gaze point data from the initial eye movement feature data, and at the same time correct the saccade focus based on the degree of astigmatism to obtain the eye movement feature data.

[0129] Removing abnormal gaze point data may include:

[0130] Eye tracking equipment is used to obtain the direction and angle of the subject's astigmatism axis. Based on the direction of the astigmatism axis, the gaze point data is divided into different directional subsets. The data from each subset is analyzed using statistical methods, such as calculating the standard deviation of the gaze point position, to set a reasonable threshold range, also known as the preset threshold range. If the gaze point position exceeds this preset threshold range, it is identified as abnormal and removed. This method improves the accuracy of the gaze point data.

[0131] Correcting the focus of eye movements can be achieved in the following ways:

[0132] A saccade focus correction model is established based on astigmatism. For example, for every 100-degree increase in astigmatism, the saccade endpoint shifts perpendicular to the astigmatism axis by a certain distance, determined through experimental or clinical data. After obtaining the subject's astigmatism, the saccade focus in the initial eye movement feature data is corrected according to the model to optimize the data.

[0133] For subjects with glaucoma, it is necessary to complete the area where the gaze point is missing in the initial eye movement feature data, and based on the relationship between the saccade data and the area where the gaze point is missing, correct the saccade trajectory. At the same time, based on the severity of the disease, adjust the scanning speed to obtain eye movement feature data.

[0134] Among them, the following methods can be used to complete the fixation area and correct the eye saccade trajectory:

[0135] Determine the subject's visual field defect area. If the initial eye movement feature data indicates a missing fixation point in the missing area, use an interpolation algorithm, such as linear or spline interpolation, to estimate and complete the missing fixation point based on the distribution of fixations around the missing area. Analyze the relationship between the saccade data and the missing fixation point area. If a saccade passes through the missing area, modify the saccade trajectory based on the start and end points of the saccade and normal eye movement patterns to ensure that the saccade trajectory conforms to normal visual behavior.

[0136] Regarding the correction process for saccadic velocity, glaucoma can be categorized as mild, moderate, or severe based on the severity of the condition. Clinical studies have been conducted to determine the patterns of saccadic velocity changes in patients with varying degrees of glaucoma. Based on the severity of the subject's condition and the patterns of saccadic velocity changes, the saccadic velocity in the initial eye movement feature data is adjusted accordingly to improve the eye movement feature data.

[0137] For subjects with cataracts, the eye movement velocity in the initial eye movement feature data needs to be adjusted and abnormal gaze point data needs to be removed; the rotation frequency data needs to be adjusted based on the subject's average blink frequency to obtain eye movement feature data.

[0138] In this embodiment, adjusting eye movement velocity and removing abnormal fixations can be achieved by:

[0139] The eye movement velocity adjustment ratio is set based on the severity of the cataract. Based on this adjustment ratio and the severity of the cataract, the eye movement velocity in the initial eye movement feature data is adjusted accordingly. Using a similar method to astigmatism, the standard deviation of the fixation point position is calculated to set a threshold, remove abnormal fixation points, and improve the quality of the eye movement feature data.

[0140] Adjusting blink rate data can be achieved in the following ways:

[0141] Calculate the subject's average blink frequency over a period of time. Based on the severity of the cataract, adjust the average blink frequency data in the initial eye movement feature data accordingly to obtain the final eye movement feature data.

[0142] In summary, adjusting data through eye diseases can improve the accuracy of eye movement data and enhance interaction accuracy. Eye control commands determined based on accurate data are more in line with the patient's intentions and avoid misjudgments in human-computer interaction.

[0143] In an optional embodiment, determining the subject's eye control command based on brain activity state information, eye movement feature data, and user feature data includes:

[0144] If the brain activity state information is normal, the correlation priority between each feature data in the user feature data and the eye control command is determined.

[0145] The subject's eye control instructions are determined based on the relevance priority of each feature data and the eye movement feature data.

[0146] In this embodiment, because different eye movement feature data is set for different users, more than one eye control command may be matched when determining an eye control command. To avoid execution errors, the correlation priority between each feature data in the user feature data and the eye control command can be determined. Based on the determined correlation priority of each feature data, the eye control command to be executed first among multiple eye control commands can be determined.

[0147] In an optional embodiment, determining the subject's eye control instruction based on the correlation priority of each feature data and the eye movement feature data includes:

[0148] Each feature data and eye movement feature data are matched with the rules in the preset rule library to obtain the target eye control instruction.

[0149] The subject's eye control instructions are determined based on the relevance priority of each feature data.

[0150] In this embodiment, the preset rule base is a database that stores the relationships between various feature data combinations and corresponding eye control commands. Corresponding eye control commands are defined based on different feature data combinations. For example, for a person aged 20-30, with brown pupil color, no eye disease, and eye movement characteristics characterized by rapid saccades, the corresponding eye control command is defined as "switch page."

[0151] The subject's various user feature data and eye movement feature data are input into the rules in the preset rule library to find the corresponding target eye control instructions. If there are multiple target eye control instructions, the target eye control instruction corresponding to the user feature data with the highest priority will be used as the subject's eye control instruction based on the priority of the various user feature data; or, if there is only one target eye control instruction determined based on the eye movement feature data, but this instruction conflicts with the instruction matched according to the type of eye disease, the instruction matched by the eye movement feature data shall prevail.

[0152] In an optional embodiment, determining the relevance priority between each feature data in the user feature data and the eye control instruction includes:

[0153] Analyze the correlation between various features in user feature data and eye control commands.

[0154] According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0155] In this embodiment, a correlation coefficient method may be used to analyze the correlation between each feature in the user feature data and the eye control command, so as to obtain the correlation priority between each feature data of different users and the eye control command.

[0156] In an optional embodiment, determining the subject's eye control command based on brain activity state information, eye movement feature data, and user feature data includes:

[0157] If the brain activity state information is: the first abnormal situation; then based on the subject's historical interaction data, predict the area that the subject may currently focus on, and update the area where the gaze point is located based on the prediction result; and subtract the two characteristic data of adaptation time and gaze time when switching between different targets from the preset error tolerance time, respectively, to obtain the corrected adaptation time and updated gaze time; thereby obtaining new eye movement characteristic data; the error tolerance time can be determined based on the subject's adaptation time and gaze time and the adaptation time and gaze time of a normal person; for example, if the subject's adaptation time is 1.2 seconds and the normal person's adaptation time is less than 500 milliseconds, the error tolerance time can be set to 700 milliseconds;

[0158] If the brain activity state information is: the second abnormal situation; then obtain the subject's gaze point movement trajectory, filter the gaze point movement trajectory based on the adaptive Kalman filter or sliding average filter method, dynamically filter the high-frequency chattering noise, obtain the true gaze point movement trajectory, and correct the eye movement feature data based on the true gaze point movement trajectory to obtain new eye movement feature data;

[0159] The subject's eye control instructions are determined based on the new eye movement feature data and user feature data.

[0160] It should be noted that the method of determining the subject's eye control instructions based on the new eye movement feature data and user feature data in this embodiment is the same as the method in the above embodiment, and will not be repeated here.

[0161] In this embodiment, if the brain activity state information indicates that different subjects may have different abnormal brain activity states (such as inattention, frequent switching of gaze points, slow reaction speed, etc.), the eye movement characteristics affected by different abnormal conditions are corrected to obtain new eye movement characteristics. The eye control instructions generated based on the new eye movement feature data and user feature data compensate for the deviation of the eye movement characteristics caused by the abnormal brain activity state, so that the eye control instructions finally generated can accurately reflect the subject's true intention and improve the accuracy of eye control interaction.

[0162] In summary, the embodiments of the present invention consider the characteristics of different user types, collect multiple user feature data, and perform feature extraction based on eye movement data. This generates eye movement feature data that accurately reflects the user's true eye movement intentions. This data can avoid command misjudgments caused by ignoring individual differences, significantly improving the accuracy of determining eye control commands and making interactions more accurate. When determining eye control commands, the characteristics of different users are further considered to adapt the resulting eye control commands to each user, improving the accuracy of eye control interactions while providing users with a smoother and more natural interactive experience.

[0163] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0164] Figure 4 This is a schematic diagram of the structure of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention.

[0165] like Figure 4 The eye-tracking-based eye-controlled interaction device provided in this embodiment includes:

[0166] Data collection module 401 is used to collect eye movement data and user characteristic data of the subject; wherein the user characteristic data includes age, pupil color and type of eye disease;

[0167] A feature extraction module 402 is used to extract eye movement feature data of the eye movement data based on the eye movement data and the user feature data;

[0168] The information generation module 403 is used to generate the subject's brain activity state information based on the eye movement feature data;

[0169] An instruction generation module 404 is configured to determine an eye control instruction of the subject based on the brain activity state information, the eye movement feature data, and the user feature data;

[0170] The eye control interaction module 405 is configured to perform eye control interaction based on the eye control instruction.

[0171] In a possible implementation, the feature extraction module 402 is specifically configured to determine pupil light reflectance according to the pupil color of the subject;

[0172] enhancing the pupil boundary in the eye movement data according to the pupil light reflectivity;

[0173] determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data;

[0174] Determine the impact data based on the type of eye disease in question;

[0175] Based on the impact data and the initial eye movement feature data, eye movement feature data is obtained

[0176] In a possible implementation, the feature extraction module 402 is further configured to use the saccade amplitude and frequency as extraction indicators if the age of the subject is within a first preset age range;

[0177] If the age of the subject is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators;

[0178] If the age of the subject is within the third preset age range, blink frequency, adaptation time when switching gazes to targets at different distances, gaze point stability, saccade speed and gaze time are used as extraction indicators.

[0179] In a possible implementation, the feature extraction module 402 is further configured to, if the eye disease type is myopia, have the influencing data be the saccade amplitude and frequency and the adaptation time when switching gazes between targets at different distances;

[0180] If the eye disease type is hyperopia, the impact data are saccade amplitude, frequency, and fixation time;

[0181] If the eye disease type is astigmatism, the influencing data are the saccade amplitude, frequency, and gaze point stability;

[0182] If the eye disease type is glaucoma, the influencing data are the saccade amplitude, frequency, scanning speed, and the area where the gaze point is located;

[0183] If the eye disease type is cataract, the impact data are eye movement speed, gaze point stability, and blink frequency.

[0184] In one possible implementation, the feature extraction module 402 is further configured to, if the eye disease type is myopia, remove abnormal saccade data from the initial eye movement feature data based on the subject's historical eye movement data and predicted eye movement data at the next moment, and adjust the adaptation time when switching gazes between targets at different distances according to the degree of myopia, so as to obtain eye movement feature data.

[0185] If the eye disease type is hyperopia, then based on the average mixed eye movement data of the subject, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced, and the fixation time is corrected based on the degree of hyperopia to obtain eye movement feature data;

[0186] If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, removing abnormal gaze point data from the initial eye movement feature data, and simultaneously correcting the saccade focus based on the degree of astigmatism to obtain eye movement feature data;

[0187] If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the saccade trajectory is corrected based on the relationship between the saccade data and the missing fixation point area. At the same time, the saccade speed is adjusted based on the severity of the disease to obtain eye movement feature data;

[0188] If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and abnormal gaze point data is removed; the rotation frequency data is adjusted based on the average blink frequency of the subject to obtain eye movement feature data.

[0189] In one possible implementation, the instruction generation module 404 is specifically configured to: if the brain activity state information is normal, determine a correlation priority between each feature data in the user feature data and the eye control instruction;

[0190] The eye control instruction of the subject is determined according to the relevance priority of each feature data and the eye movement feature data.

[0191] In a possible implementation, the instruction generation module 404 is specifically configured to: match the various feature data and the eye movement feature data with rules in a preset rule library to obtain a target eye control instruction;

[0192] The eye control instruction of the subject is determined according to the relevance priority of each feature data.

[0193] In a possible implementation, the instruction generation module 404 is further configured to:

[0194] Analyze the correlation between various features in the user feature data and eye control commands;

[0195] According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0196] Figure 5 Schematic diagram of the structure of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention.

[0197] like Figure 5 As shown, the eye-controlled interaction device 102 includes a memory 1021 and a processor 1022. The memory 1021 stores a computer program, and the processor 1022 implements the steps of the above-mentioned method embodiments when executing the computer program. Alternatively, the processor 1022 implements the functions of the modules / units in the above-mentioned device embodiments when executing the computer program.

[0198] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 1021 and executed by the processor 1022 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device 102.

[0199] The eye-controlled interaction device 102 may include, but is not limited to, a processor 1022 and a memory 1021. Those skilled in the art will appreciate that Figure 5 This is only an example of the eye-controlled interaction device 102 and does not constitute a limitation of the eye-controlled interaction device 102. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the eye-controlled interaction device 102 may also include input and output devices, network access devices, buses, etc.

[0200] The processor 1022 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0201] The memory 1021 can be an internal storage unit of the eye-controlled interaction device 102, such as the hard drive or memory of the eye-controlled interaction device 102. The memory 1021 can also be an external storage device of the eye-controlled interaction device 102, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the eye-controlled interaction device 102. Furthermore, the memory 1021 can include both the internal storage unit of the eye-controlled interaction device 102 and an external storage device. The memory 1021 is used to store computer programs and other programs and data required by the eye-controlled interaction device 102. The memory 1021 can also be used to temporarily store data that has been output or is about to be output.

[0202] For the sake of convenience and brevity, the division of the above functional modules / units is only used as an example. In actual applications, the above functions can be assigned to different functional modules / units as needed. The above modules / units can be implemented in the form of hardware, software, or a combination of hardware and software.

[0203] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in the above-mentioned method embodiments.

[0204] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the methods in the above-mentioned method embodiments.

[0205] The term "computer program" includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media.

[0206] In the above embodiments, the descriptions of each embodiment have their own focus. For parts not described or recorded in detail in one embodiment, please refer to the relevant descriptions of other embodiments. Unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features of different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0207] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An eye-controlled interaction method based on eye tracking, characterized in that: include: Collecting eye movement data and user characteristic data of the subject; wherein the user characteristic data includes age, pupil color, and type of eye disease; extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data; generating brain activity state information of the subject based on the eye movement feature data; determining an eye control instruction of the subject according to the brain activity state information, the eye movement feature data, and the user feature data; Performing eye control interaction based on the eye control instruction; Extracting eye movement feature data of the eye movement data based on the eye movement data and the user feature data includes: determining pupil light reflexivity based on the pupil color of the subject; enhancing the pupil boundary in the eye movement data according to the pupil light reflectivity; determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data; Determine the impact data based on the type of eye disease in question; Eye movement feature data is obtained based on the impact data and the initial eye movement feature data.

2. The eye-tracking-based eye-controlled interaction method according to claim 1, characterized in that: The step of determining the extraction index according to the age of the subject includes: If the subject's age is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators; If the age of the subject is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators; If the age of the subject is within the third preset age range, blink frequency, adaptation time when switching gazes to targets at different distances, gaze point stability, saccade speed and gaze time are used as extraction indicators.

3. The eye-tracking-based eye-controlled interaction method according to claim 1, characterized in that: Determining the impact data according to the type of eye disease includes: If the eye disease type is myopia, the influencing data are the amplitude and frequency of saccades and the adaptation time when switching gazes to targets at different distances; If the eye disease type is hyperopia, the impact data are saccade amplitude, frequency, and fixation time; If the eye disease type is astigmatism, the influencing data are the saccade amplitude, frequency, and gaze point stability; If the eye disease type is glaucoma, the influencing data are the saccade amplitude, frequency, scanning speed, and the area where the gaze point is located; If the eye disease type is cataract, the impact data are eye movement speed, gaze point stability, and blink frequency.

4. The eye-tracking-based eye-controlled interaction method according to claim 3, characterized in that: The obtaining of eye movement feature data based on the impact data and the initial eye movement feature data includes: If the eye disease type is myopia, then based on the subject's historical eye movement data and the predicted value of the eye movement data at the next moment, abnormal eye movement data in the initial eye movement feature data is removed, and at the same time, the adaptation time when switching gaze between targets at different distances is adjusted according to the degree of myopia to obtain eye movement feature data; If the eye disease type is hyperopia, then based on the average mixed eye movement data of the subject, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced, and the fixation time is corrected based on the degree of hyperopia to obtain eye movement feature data; If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, removing abnormal gaze point data from the initial eye movement feature data, and simultaneously correcting the saccade focus based on the degree of astigmatism to obtain eye movement feature data; If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the saccade trajectory is corrected based on the relationship between the saccade data and the missing fixation point area. At the same time, the saccade speed is adjusted based on the severity of the disease to obtain eye movement feature data; If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and abnormal gaze point data is removed; the rotation frequency data is adjusted based on the average blink frequency of the subject to obtain eye movement feature data.

5. The eye-tracking-based eye-controlled interaction method according to claim 1, characterized in that: The determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data, and the user feature data includes: If the brain activity state information is normal, determining the correlation priority between each feature data in the user feature data and the eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data and the eye movement feature data.

6. The eye-tracking-based eye-controlled interaction method according to claim 5, characterized in that: The determining of the eye control instruction of the subject according to the correlation priority of each feature data and the eye movement feature data includes: Matching the various feature data and the eye movement feature data with the rules in a preset rule library to obtain a target eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data.

7. The eye-tracking-based eye-controlled interaction method according to claim 5, characterized in that: Determining the relevance priority between each feature data in the user feature data and the eye control instruction includes: Analyze the correlation between various features in the user feature data and eye control commands; According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

8. An eye-tracking-based eye-controlled interactive device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

9. An eye-tracking-based eye-controlled interaction system, characterized in that: An eye tracker and an eye-controlled interaction device based on eye tracking as claimed in claim 8.

Images