Eye control interaction method, device and system based on eye movement tracking

Abstract

The invention provides an eye control interaction method, device and system based on eye movement tracking, and relates to the technical field of data processing. The method comprises the following steps: collecting eye movement data and user feature data of a testee; the user feature data comprises age, pupil color and eye disease type; extracting eye movement feature data according to the eye movement data and the user feature data; generating brain activity state information of the testee according to the eye movement feature data, and determining an eye control instruction of the testee according to the brain activity state information, the eye movement feature data and the user feature data; and performing eye control interaction based on the eye control instruction. According to the method, the characteristics of brain activity states, eye diseases and the like possibly existing in different types of users can be considered, the user feature data are collected, feature extraction is carried out in combination with the eye movement data, and the eye movement feature data accurately reflecting the real eye movement intention of the user is obtained, so that the obtained eye control instruction is adaptive to different users; while the eye control interaction accuracy is improved, smoother and more natural interaction experience is brought to the user.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an eye-control interaction method, device and system based on eye tracking. Background Art

[0002] Eye tracking technology has developed rapidly in the fields of human-computer interaction, medical and engineering integration, and has become a key means to enhance the naturalness and immersion of interaction. In recent years, with the rise of virtual reality (VR) and augmented reality (AR) technologies, the application scenarios of eye tracking technology have continued to expand. For example, eye tracking technology is increasingly used in game design. It can not only enhance the immersion and interactive experience of the game, but also help developers optimize game design and improve user experience. In addition, eye tracking technology is increasingly used in neurology (neurology) and medical fields, especially in the early assessment of neurodegenerative diseases and cognitive functions.

[0003] In the related art, the method of realizing human-computer interaction based on eye tracking technology is usually: the human-computer interaction device obtains the human's eye movement data, extracts eye movement features from the eye movement data, generates corresponding eye control commands by identifying the eye movement features, and realizes human-computer interaction through the eye control commands.

[0004] Although eye tracking technology has shown great potential in the fields of human-computer interaction, medical and engineering integration, it still faces many challenges. This challenge is mainly due to the differences between different users: on the one hand, eye movement data is regulated by the brain, and some users may have abnormal brain activity (such as abnormal attention allocation, inhibitory control ability, etc.), their limb movement behavior is normal but the eye movement data will have slight abnormalities; on the other hand, some users have eye diseases that can also cause abnormal eye movement data, etc.; this difference will cause the eye movement features extracted by eye tracking technology, the eye control commands recognized and generated to be inconsistent with the actual intentions that the user actually wants to express, and the calibration function of the eye tracking device at this stage is difficult to accurately adjust the relevant control commands for different users, resulting in low recognition accuracy and difficulty in adapting to the diverse needs of users and scenarios. Summary of the invention

[0005] The embodiments of the present invention provide an eye-controlled interaction method, device and system based on eye tracking to solve the problem of low recognition accuracy of eye tracking during the interaction process at this stage.

[0006] In a first aspect, an embodiment of the present invention provides an eye-controlled interaction method based on eye tracking, comprising: Collecting the subject's eye movement data and user characteristic data; wherein the user characteristic data includes age, pupil color, and type of eye disease; Extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data; generating brain activity state information of the subject according to the eye movement feature data; determining eye control instructions of the subject according to the brain activity state information, the eye movement feature data and the user feature data; Perform eye-controlled interaction based on eye-controlled commands.

[0007] In a possible implementation, extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data includes: The pupil light reflex was determined based on the subject's pupil color; The pupil boundary in the eye movement data is enhanced according to the pupil light reflex; Determine the extraction index according to the age of the subject; and extract features from the eye movement data based on the extraction index to obtain initial eye movement feature data; Determine impact data based on the type of eye disease; Eye movement feature data is obtained based on the impact data and the initial eye movement feature data.

[0008] In a possible implementation, the extraction index is determined according to the age of the subject, including: If the subject's age is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators; If the subject's age is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed, and visual search efficiency are used as extraction indicators; If the subject's age is within the third preset age range, blink frequency, adaptation time when switching gazes between targets at different distances, gaze point stability, saccade speed, and gaze time are used as extraction indicators.

[0009] In a possible implementation, the impact data is determined according to the type of eye disease, including: If the eye disease type is myopia, the affected data are the amplitude and frequency of saccades and the adaptation time when switching gaze to targets at different distances; If the eye disease type is hyperopia, the affected data are saccade amplitude, frequency, and fixation time; If the eye disease type is astigmatism, the affected data are the amplitude and frequency of saccades and the stability of the fixation point; If the eye disease type is glaucoma, the affected data are the saccade amplitude, frequency, saccade speed, and the area where the fixation point is located; If the type of eye disease is cataract, the affected data are eye movement speed, gaze point stability, and blinking frequency.

[0010] In a possible implementation, the eye movement feature data is obtained based on the impact data and the initial eye movement feature data, including: If the eye disease type is myopia, then based on the subject's historical eye movement data and the predicted value of the eye movement data at the next moment, the abnormal eye saccade data in the initial eye movement feature data is removed, and at the same time, according to the degree of myopia, the adaptation time when switching gazes to targets at different distances is adjusted to obtain the eye movement feature data; If the eye disease type is hyperopia, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced based on the average mixed eye movement data of the subject, and the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data; If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, the abnormal fixation point data in the initial eye movement feature data is removed, and the focus of the eye saccade is corrected based on the degree of astigmatism to obtain the eye movement feature data; If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the eye saccade trajectory is corrected based on the relationship between the eye saccade data and the missing fixation point area. At the same time, the scanning speed is adjusted based on the severity of the disease to obtain the eye movement feature data; If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the turn frequency data is adjusted based on the average blinking frequency of the subject to obtain the eye movement feature data.

[0011] In a possible implementation, determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data includes: If the brain activity state information is normal, determining the correlation priority between each feature data in the user feature data and the eye control command; The eye control instructions of the subject are determined according to the relevance priority of each feature data and the eye movement feature data.

[0012] In a possible implementation, the eye control instruction of the subject is determined according to the relevance priority of each feature data and the eye movement feature data, including: Match each feature data and eye movement feature data with the rules in the preset rule library to obtain the target eye control instruction; The subject's eye control instructions are determined according to the relevance priority of each feature data.

[0013] In a possible implementation, determining the correlation priority between each feature data in the user feature data and the eye control instruction includes: Analyze the correlation between various features in the user feature data and the eye control commands; According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0014] In a second aspect, an embodiment of the present invention provides an eye-controlled interaction device based on eye tracking, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method in the first aspect or any possible implementation method of the first aspect is implemented.

[0015] In a third aspect, an embodiment of the present invention provides an eye-tracking-based eye-controlled interaction system, an eye tracker, and an eye-tracking-based eye-controlled interaction device as provided in the second aspect above.

[0016] In the embodiment of the present invention, the characteristics of different types of users are taken into consideration, multiple user feature data are collected and combined with eye movement data for feature extraction, and eye movement feature data that can accurately reflect the user's true eye movement intention is obtained. These data can avoid misjudgment of instructions due to ignoring individual differences, greatly improve the accuracy of determining eye control instructions, and make the interaction more accurate. When determining the eye control instruction, brain activity state information is generated based on the eye movement feature data to determine the user's brain health state (such as whether there are abnormal conditions such as inattention, frequent switching of gaze points, inability to accurately gaze at the target area, and slow reaction speed), and eye control instructions are generated based on brain activity state information, user feature data, and eye movement feature data. The characteristics of different users are taken into consideration again, so that the obtained eye control instructions are adapted to different users, improving the accuracy of eye control interaction while providing users with a smoother and more natural interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a system architecture diagram of an eye-control interaction system based on eye tracking provided by an embodiment of the present invention; Figure 2 is a flow chart of an implementation of an eye-control interaction method based on eye tracking provided by an embodiment of the present invention; Figure 3 is a flowchart for extracting eye movement feature data provided by an embodiment of the present invention; Figure 4 is a schematic diagram of the structure of an eye-control interaction device based on eye tracking provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the related art, in the process of realizing human-computer interaction based on eye tracking technology, the same eye movement features are usually extracted for different users, and then eye control instructions are generated based on the extracted eye movement features, and human-computer interaction is realized according to the eye control instructions. However, there may be differences between different users, and this difference is mainly reflected in the differences in physiological factors and pathological factors. Pathological factors mainly include abnormal brain activity state and eye diseases: on the one hand, eye movement data is regulated by the brain, and some users may have abnormal brain activity state (such as whether there is inattention and frequent switching of gaze points, inability to accurately focus on the target area, and slow reaction speed, etc.), their limb movement behavior is normal but eye movement data will have slight abnormalities; on the other hand, some users have eye diseases (such as myopia, glaucoma, etc.), which will also cause abnormal eye movement data, etc.; physiological factors mainly include pupil characteristics and age. These differences will cause the eye movement features extracted by eye tracking technology, the eye control instructions identified and generated, to be inconsistent with the actual intentions that the user actually wants to express, and the calibration function of the eye tracking device at this stage is difficult to accurately adjust the relevant control instructions for different users, resulting in low recognition accuracy and difficulty in adapting to diverse user and scene requirements.

[0020] In response to the above defects, the present application proposes the following technical ideas: when extracting eye movement feature data, the characteristics of different types of users are taken into consideration, and multiple user feature data (including age, pupil color, and type of eye disease) are collected and combined with eye movement data for feature extraction to obtain eye movement feature data that can accurately reflect the user's true eye movement intentions. These data can avoid misjudgment of commands due to ignoring individual differences, greatly improve the accuracy of determining eye control commands, and make the interaction more accurate. When determining eye control commands, brain activity state information is generated based on eye movement feature data to determine the user's brain health state (such as whether there are abnormal conditions such as inattention, frequent switching of gaze points, inability to accurately focus on the target area, and slow reaction speed), and eye control commands are generated based on brain activity state information, user feature data, and eye movement feature data. The characteristics of different users are taken into consideration again, so that the obtained eye control commands are adapted to different users, improving the accuracy of eye control interaction while providing users with a smoother and more natural interaction experience.

[0021] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 The system architecture diagram of the eye-tracking-based eye-controlled interaction system provided by the embodiment of the present invention is shown in FIG. Figure 1 As shown, the system may include an eye tracker 101 and an eye control interaction device 102 based on eye tracking. When applied, the eye movement data of the subject can be collected by the eye tracker 101, and then the eye movement data can be transmitted to the eye control interaction device 102 based on eye tracking, and the eye control interaction device 102 based on eye tracking can analyze the eye movement data to obtain eye control instructions. The corresponding screen will be displayed in the eye tracker. When the eye control instruction is controlling, the control process can be displayed in the eye tracker, so that the subject can determine whether the recognized eye control instruction is correct, and if it is incorrect, it can be adjusted in time.

[0023] In this embodiment, the eye tracker can be any of the three forms: open, desktop, and vertical. The three forms of eye trackers have the same components, working principles and mechanisms, but different integration methods. Each eye tracker can include a host, an acquisition device, and a display device. In actual use, different eye trackers can be deployed and installed according to user needs.

[0024] The following examples illustrate the installation process of eye trackers of different forms: Exemplarily, the installation process of an open-type eye tracker is as follows: prepare a socket with more than 4 sockets, a countertop for placing the subject's monitor, etc., and a desktop for placing the monitor, etc.; install the subject's monitor, monitor, computer, camera and light source, and head and jaw support; connect the data transmission according to the instructions; connect the power cords of all devices that need power to the socket; conduct an acceptance test, start the eye tracker software after booting into the system, and check whether the video image and various functions are normal.

[0025] The installation process of a desktop eye tracker is as follows: prepare a socket with more than 4 ports, a table for placing desktop devices, a desktop for placing monitors, etc.; install desktop devices, monitors, and computers; connect the camera data cable and data transmission cable according to the instructions; connect the power cords of all devices that need power to the socket; conduct an acceptance test, start the eye tracker software after booting into the system, and check whether the video image and various functions are normal.

[0026] The installation process of a vertical eye tracker is as follows: prepare a socket with more than 4 ports, a desktop for placing a monitor, and space for installing vertical equipment; install vertical equipment, such as releasing the universal wheel brake device and the monitor; connect the equipment and data transmission lines according to the instructions; connect the power cords of all devices that need power to the socket; perform an acceptance test, start the eye tracker software after booting into the system, and check whether the video image and various functions are normal.

[0027] In this embodiment, the eye-tracking-based eye-controlled interaction device 102 includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements an eye-movement data analysis method to obtain eye-control instructions for eye-controlled interaction.

[0028] Alternatively, an embodiment of the present invention may also be implemented based on a near-eye display device, which includes an eye-control interaction device 102 based on eye tracking. The eye-control interaction device 102 based on eye tracking collects eye movement data through the near-eye display device, and obtains eye control instructions based on the collected near-eye display device and user feature data, so that the near-eye display device implements eye control interaction based on the eye control instructions.

[0029] Figure 2 is a flowchart of an eye control interaction method based on eye tracking provided by an embodiment of the present invention. Figure 2 The method implemented by the eye-tracking-based eye-controlled interaction device 102 is described below: Step S201: collecting the subject's eye movement data and user characteristic data; wherein the user characteristic data includes age, pupil color and eye disease type.

[0030] In this embodiment, the eye movement data of the subject can be collected by an eye tracker or a near-eye display device.

[0031] For example, the eye movement data can be obtained through the eye tracker's specific interface function and presented in the form of an array. Each row represents different eye movement information, including the frame number, the X and Y coordinates of the left and right pupils, the pupil diameter, the area, the X and Y coordinates of the left and right eye reflection points, and the X and Y coordinates of the left and right eye viewing angles. These parameters accurately record key information such as the position of the eyes, pupil size, and viewing angle direction in each frame, providing a detailed data basis for subsequent eye movement analysis.

[0032] Alternatively, eye movement data can also be stored in the form of videos and images through an eye tracker. Video files intuitively show the state of the eyes at different times. By analyzing videos and images, we can observe information such as the movement trajectory of the eyes and the change process of the pupils, which complements the data in the form of arrays and provides researchers with a more comprehensive perspective on eye movement data.

[0033] In this embodiment, the user characteristic data may be information entered by the subject during use, which may include the user's age, pupil color, and type of eye disease; wherein eye diseases may include normal, myopia, hyperopia, astigmatism, glaucoma, cataracts, etc.

[0034] Step S202: extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data.

[0035] In this embodiment, the traditional method extracts common eye movement features for different users. However, due to differences in age, pupil color, and eye diseases among different users, the same features are not sufficient to accurately characterize the intentions of each user. Therefore, the method in this embodiment fully considers the differences among different users in terms of age, pupil color, and type of eye diseases. These user feature data provide an important basis for in-depth analysis of eye movement data.

[0036] People's eye movement characteristics vary with age. Young people may have faster eye movement reaction speed, while older people's eye movement speed is relatively slow and their gaze stability may be poor. By incorporating age factors, eye movement features can be extracted more accurately for users of different age groups.

[0037] Different pupil colors reflect different light, which will affect the collection and analysis of eye movement data. For example, lighter pupils may reflect more light under the same lighting conditions, resulting in different characteristics of the collected eye movement data.

[0038] The type of eye disease directly affects the eye movement pattern. Different diseases such as myopia, hyperopia, astigmatism, glaucoma, and cataracts will cause eye movements to show different characteristics in terms of amplitude, frequency, and fixation time. Extracting eye movement features by combining these user feature data can fully reflect the eye movement characteristics of different users and avoid analysis errors caused by ignoring individual differences.

[0039] Step S203: Generate the subject's brain activity state information based on the eye movement feature data.

[0040] In this step, the eye movement feature data may include but is not limited to: saccade amplitude, saccade frequency, adaptation time when switching gaze to targets at different distances, gaze time, gaze point stability, scan rate, gaze point location, and blinking frequency.

[0041] Since the process of eye movement is coordinated and controlled by multiple areas of the brain and cerebellum (including the frontal lobe eye movement area, parietal lobe, basal ganglia, thalamus, brainstem and other structures), the above eye movement feature data can reflect specific brain functions, such as attention allocation, visual information processing, emotional state, memory strength, inhibitory control ability and other cognitive conditions and physiological states. Therefore, based on the above eye movement feature data, it can reveal whether there is an abnormality in the subject's brain activity state (i.e., the functional state of the nervous system). For example, Parkinson's disease may have characteristics such as a smaller increase in eye saccade amplitude and frequency, a slower scan rate, and a change in blinking frequency in the early stage because of dopamine deficiency affecting eye movement control (abnormal brain activity states such as inhibitory control ability); Alzheimer's disease may have characteristics such as prolonged adaptation time, prolonged fixation time, and dispersed fixation area when switching between different targets due to impaired attention and visual processing (abnormal brain activity states such as attention allocation).

[0042] In a possible implementation, a judgment threshold range is set for each eye movement feature, the eye movement feature is compared with the corresponding judgment threshold range, and brain activity state information is generated according to the comparison result.

[0043] Exemplarily, the range of normal eye movement characteristics is usually: eye movement amplitude range is 5°~30°, eye movement frequency is 1~2 times / second in a natural state, scanning rate is 200°~400° / second, adaptation time when switching gaze between targets at different distances is usually less than 500 milliseconds, gaze time is: 200~500ms / time in the target area, gaze point stability is: jitter amplitude <1° (visual angle), the area where the gaze point is located is concentrated in the task-related interest area, and the blinking frequency is 15~20 times / minute. If all eye movement feature data are within the corresponding judgment threshold range, the generated brain activity status information is: normal; if the fixation time of 30% of the target area is less than 100 milliseconds, the area where the fixation point is located deviates from the target area (the hit rate of the interest area is less than 60%), the adaptation time when switching between different targets is greater than 1.2 seconds and the eye saccade frequency is greater than 3 times / second, the generated brain activity status information is: the first abnormal situation; if the peak rate in the scanning rate is less than 180° / second, the eye saccade amplitude in 40% of the eye saccades is less than 5°, the stability of the fixation point decreases (the jitter amplitude is greater than 1.5°), and the blinking frequency decreases (less than 12 times / minute), the generated brain activity status information is: the second abnormal situation.

[0044] Step S204: determining the subject's eye control instructions according to the brain activity state information, the eye movement feature data and the user feature data.

[0045] In this embodiment, because the association between eye movement characteristics and true intentions varies for users of different ages, pupil colors, types of eye diseases, and different brain activity states, determining eye control commands based solely on general eye movement characteristics may lead to inaccurate command interpretation and fail to meet the actual needs of diverse users. Therefore, this embodiment also takes user characteristic data into consideration when determining eye control commands.

[0046] Step S205: performing eye control interaction based on the eye control instruction.

[0047] After determining the eye control command, the corresponding interactive operations can be performed according to different application scenarios. For example, in a game scene, if the eye control command is to aim at the target, the system will accurately position the game character's perspective or crosshairs to the corresponding target based on the gaze point position determined by the eye movement data, so as to achieve rapid aiming. In a VR or AR scene, if the eye control command is to select an object in the virtual scene, the system will determine the object the user is looking at based on the eye movement data, and then trigger the corresponding selection operation, which may be accompanied by feedback such as highlighting the object and information display, so that the user clearly knows that the operation has been executed. In some smart device control scenarios, if the eye control command is to open an application, the system will recognize the eye control command and then start the corresponding application.

[0048] Figure 3 is a flowchart of eye movement feature data extraction implementation provided by an embodiment of the present invention; Figure 3 As shown, in an optional embodiment, extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data in step S202 may include: Step S2021: Determine the pupil light reflectivity according to the subject's pupil color.

[0049] Step S2022: enhancing the pupil boundary in the eye movement data according to the pupil light reflectivity.

[0050] Step S2023: determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data.

[0051] Step S2024: Determine the impact data according to the type of eye disease.

[0052] Step S2025: Obtain eye movement feature data based on the impact data and the initial eye movement feature data.

[0053] In this embodiment, the change in pupil size is an indicator of the brain's response to environmental changes and is closely related to the state of wakefulness, emotions, and cognitive functions. When extracting eye movement features, by accurately detecting the pupil center and combining other feature points such as corneal reflection points, other eye movement features such as the eye's line of sight and gaze point can be more accurately calculated.

[0054] However, different subjects have different pupil colors, which leads to different sensitivities to light. Moreover, in eye movement images, the pupil boundaries of light-colored pupils are not obvious in strong light, while the pupil boundaries of dark-colored pupils are not obvious in dark light, which leads to inaccurate recognition during extraction.

[0055] Therefore, considering the intrinsic connection between pupil color and pupil light reflectivity, generally speaking, pupils with lighter colors, such as blue pupils, have relatively higher light reflectivity, while pupils with darker colors, such as black pupils, have relatively lower light reflectivity. Based on this correlation, combined with existing research data or pre-established models, it can be determined that the light reflectivity of the subject's pupil can provide a more accurate basis for the subsequent extraction of eye movement feature data based on the enhanced boundaries, such as when analyzing the amplitude and frequency of eye saccades, it can more accurately determine changes in eye position. In the eye control operation of smart devices, the user's gaze point can be identified more accurately to achieve precise control.

[0056] In this embodiment, taking into account that different age groups understand and express intentions in different ways, the eye movement data expressed by subjects of different age groups have different characteristics. In order to ensure that the extracted features can accurately reflect the intentions of the subjects, this embodiment designs different feature extraction indicators for subjects in different age groups.

[0057] Based on the age of the subjects, extraction indicators are determined, and features are extracted from the eye movement data based on the extraction indicators to obtain initial eye movement feature data.

[0058] Then, considering that eye diseases will affect the eye movement data, the intention actually expressed by the subject will deviate from the intention detected under the influence of eye diseases. Therefore, this embodiment determines the impact of each type of eye disease on the eye movement characteristics, that is, the impact data, according to different types of eye diseases, so as to correct the initial eye movement feature data based on the impact data to obtain the eye movement feature data.

[0059] In an optional embodiment, determining the extraction index according to the age of the subject in step S2023 may include: If the subject's age is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators.

[0060] If the subject's age is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators.

[0061] If the subject's age is within the third preset age range, blink frequency, adaptation time when switching gazes between targets at different distances, gaze point stability, saccade speed, and gaze time are used as extraction indicators.

[0062] In this embodiment, according to the characteristics of the age group, this embodiment sets three age intervals, wherein the first preset age interval represents the child stage, the second preset age interval represents the adolescent stage, and the third preset age interval represents the adult stage. The first preset age interval may be 0-12 years old, the second preset age interval may be 13-18 years old, and the third preset age interval may be 19 years old and above. Of course, in actual application, it can also be adjusted according to actual needs.

[0063] In the childhood stage, since children understand and express their intentions differently from adults, their eye movements may be more random. When determining eye control instructions, considering their easily distracted and frequent eye saccades, multiple short-term gazes at the same area can be determined as an operational intention, and the instruction setting is relatively simple and direct. For example, in educational games, multiple gazes at a knowledge point icon in a short period of time can be determined as an eye control instruction to obtain a detailed explanation of the knowledge point. Therefore, for subjects in this age group, the amplitude and frequency of eye saccades can be used as extraction indicators, and accordingly, the initial eye movement feature data can be data that can reflect the amplitude and frequency of eye saccades.

[0064] Adolescents have active minds and their visual systems are approaching maturity, but they have unique strategies in complex tasks. Indicators such as the first fixation time on the target, the length of the search path, and the number of repeated fixations can be extracted. In addition, considering the changes in adolescents' concentration during long-term tasks, the analysis of the distribution of fixation time is added, such as statistical changes in the length of fixation time at different task stages. Therefore, for subjects in this age group, the amplitude, frequency, fixation time, scanning speed, and visual search efficiency of eye movements can be used as extraction indicators. Correspondingly, the initial eye movement feature data can be data that can reflect the amplitude, frequency, fixation time, scanning speed, and visual search efficiency of eye movements.

[0065] Adults will experience fatigue during long-term visual tasks, so it is possible to extract changes in gaze stability, such as the frequency of small movements of the gaze point, the increase in blinking frequency, the smoothness of the start and end of saccades, such as the change in acceleration during saccades, and visual search efficiency in complex scenes. Therefore, for subjects in this age group, blinking frequency, adaptation time when switching gazes to targets at different distances, gaze point stability, saccade speed, and gaze time can be used as extraction indicators. Accordingly, the initial eye movement feature data can be data that can reflect blinking frequency, adaptation time when switching gazes to targets at different distances, gaze point stability, saccade speed, and gaze time.

[0066] In an optional embodiment, determining the impact data according to the eye disease type in step S2024 may include: If the type of eye disease is myopia, the affected data are the amplitude and frequency of saccades and the adaptation time when switching gaze to targets at different distances.

[0067] If the type of eye disease is hyperopia, the affected data are the amplitude, frequency and fixation time of saccades.

[0068] If the type of eye disease is astigmatism, the affected data are the amplitude and frequency of saccades and the stability of the gaze point.

[0069] If the type of eye disease is glaucoma, the affected data are the saccade amplitude, frequency, scanning speed and the area where the gaze point is located.

[0070] If the type of eye disease is cataract, the affected data are eye movement speed, gaze point stability, and blinking frequency.

[0071] In this embodiment, if the subject does not have an eye disease, that is, the eye disease type is healthy, the impact data is empty, or the impact data does not need to be determined, and the eye control command is determined based on the initial eye movement feature data and the user feature data.

[0072] If the subject suffers from myopia, myopia will cause changes in the refractive power of the eye and affect the eye movement pattern. Since the axial length of myopic patients is longer, it may affect the amplitude and frequency of eye saccades, and when reading or operating at close range, the gaze point may be closer to the target. Therefore, in the case of myopia, it will affect indicators such as the amplitude and frequency of eye saccades and the adaptation time when switching gazes between targets at different distances.

[0073] If the subject suffers from hyperopia, the hyperopic eye needs stronger adjustment when looking at close objects, which will be reflected in the eye movement. Hyperopic patients have difficulty adjusting when looking at close objects, and are prone to unstable gaze points and abnormal eye movements. Therefore, in the case of hyperopia, it will affect indicators such as eye movement amplitude, frequency, and fixation time.

[0074] If the subject suffers from astigmatism, the astigmatism will make the refractive power of the eyes different in different directions, resulting in differences in eye movement characteristics in different directions. There will be errors in the accuracy of eye saccades and the stability of the gaze point of such subjects. Therefore, for such subjects, the indicators such as eye saccade amplitude, frequency and gaze point stability will be affected.

[0075] If the subject suffers from glaucoma, it will affect the function of the optic nerve, leading to visual field defects and abnormal eye movements. For such subjects, there is a defect in the area where their gaze point is located, and it will also affect the amplitude, frequency, and scanning speed of eye saccades.

[0076] If the subject suffers from cataracts, the cataracts will cause the lens to become cloudy, affecting vision, resulting in slower eye movement speed and reduced gaze point stability. Therefore, for such subjects, eye movement speed, gaze point stability, and blinking frequency can be used as influencing data.

[0077] In an optional embodiment, obtaining the eye movement feature data based on the impact data and the initial eye movement feature data may include: If the type of eye disease is myopia, the abnormal saccade data in the initial eye movement feature data is removed based on the subject's historical eye movement data and the predicted value of the eye movement data at the next moment. At the same time, the adaptation time when switching gazes to targets at different distances is adjusted according to the degree of myopia to obtain eye movement feature data.

[0078] If the eye disease type is hyperopia, the saccade amplitude in the initial eye movement feature data is amplified and the saccade frequency is reduced based on the average mixed eye movement data of the subject. At the same time, the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data.

[0079] If the type of eye disease is astigmatism, the abnormal gaze point data in the initial eye movement feature data is removed based on the direction and angle of the subject's astigmatism axis, and the focus of eye saccades is corrected based on the degree of astigmatism to obtain the eye movement feature data.

[0080] If the type of eye disease is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the saccade trajectory is corrected based on the relationship between the saccade data and the missing fixation point area. At the same time, the scanning speed is adjusted based on the severity of the disease to obtain eye movement feature data.

[0081] If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the turn frequency data is adjusted based on the average blinking frequency of the subject to obtain the eye movement feature data.

[0082] In this embodiment, for subjects with myopia, it is necessary to remove abnormal eye saccade data in the initial eye movement feature data, and to correct the adaptation time when switching gazes between targets at different distances. Removing abnormal eye saccade data in the initial eye movement feature data can be achieved by the following steps: Establish a historical eye movement database of the subjects, which stores data such as eye saccade amplitude and frequency over a period of time. Use anomaly detection algorithms in time series analysis or machine learning, such as the isolation forest algorithm, to evaluate the eye saccade data in the initial eye movement feature data and identify outliers far away from most data points. Compare the predicted value of the eye movement data at the next moment with the historical data. If the difference between the predicted value and the historical data is too large and outside the abnormal range determined by the isolation forest algorithm, it is determined to be abnormal eye saccade data and removed.

[0083] Correcting the adaptation time when switching gazes at targets at different distances can be achieved through the following steps: Divide different intervals according to the degree of myopia, such as low myopia from -0.50D to -3.00D, moderate myopia from -3.25D to -6.00D, and high myopia from -6.25D and above. According to clinical research and experimental data, set a gaze adaptation time adjustment coefficient for each interval. Exemplarily, the adjustment coefficient for low myopia is 1.2, for moderate myopia is 1.5, and for high myopia is 1.8. Of course, the above data is only for illustration and can be adjusted as needed in actual applications. After the subject's myopia degree is detected, the adaptation time when switching gazes at targets at different distances initially measured is multiplied by the corresponding coefficient to obtain the adjusted adaptation time, thereby completing the optimization of the eye movement feature data.

[0084] For subjects with hyperopia, it is necessary to amplify the saccade amplitude in the initial eye movement feature data and reduce the saccade frequency. At the same time, based on the degree of hyperopia, the fixation time is corrected to obtain the eye movement feature data.

[0085] Among them, adjusting the amplitude and frequency of eye saccades can be achieved by the following steps: Calculate the average mixed eye movement data of the subjects within the preset time, including the mean of saccade amplitude and frequency. Construct a mapping relationship of saccade amplitude amplification and frequency reduction according to the degree of hyperopia. Adjust the saccade amplitude and frequency in the initial eye movement feature data according to the mapping relationship to make the eye movement features more consistent with the actual situation of hyperopic patients.

[0086] Correction of fixation time can be achieved by following the steps below: Experimental studies have found that the higher the degree of hyperopia, the longer the close-distance fixation time should be. For example, it can be set that the close-distance fixation time increases by 10% for every 1D increase in hyperopia. After obtaining the degree of hyperopia of the subject, the close-distance fixation time in the initial eye movement feature data is adjusted accordingly to complete the correction of the eye movement feature data.

[0087] For subjects with astigmatism, it is necessary to remove the abnormal fixation point data in the initial eye movement feature data, and at the same time correct the saccade focus based on the degree of astigmatism to obtain the eye movement feature data.

[0088] Wherein, removing abnormal gaze point data may include: The direction and angle information of the subject's astigmatism axis are obtained using an eye tracking device. According to the direction of the astigmatism axis, the gaze point data is divided into different direction subsets. The data of each subset is analyzed, and a statistical method is used, such as calculating the standard deviation of the gaze point position, to set a reasonable threshold range, that is, the preset threshold range. If the gaze point position exceeds the preset threshold range, it is determined as abnormal gaze point data and removed. Through the above methods, the accuracy of the gaze point data is improved.

[0089] Correcting the focus of eye saccades can be achieved in the following ways: A saccade focus correction model is established based on the astigmatism degree. For example, for every 100-degree increase in astigmatism, the saccade end point position is offset by a certain distance in the vertical direction of the astigmatism axis, and the distance is determined by experimental or clinical data. After obtaining the subject's astigmatism degree, the saccade focus in the initial eye movement feature data is corrected according to the model to optimize the eye movement feature data.

[0090] For subjects with glaucoma, it is necessary to complete the missing fixation point area in the initial eye movement feature data, and based on the relationship between the saccade data and the missing fixation point area, correct the saccade trajectory. At the same time, based on the severity of the disease, adjust the scanning speed to obtain eye movement feature data.

[0091] Among them, the following methods can be used to complete the fixation area and correct the eye saccade trajectory: Determine the defective area of ​​the subject's visual field. When it is found that the fixation point in the initial eye movement feature data is missing in the defective area, use an interpolation algorithm, such as linear interpolation or spline interpolation, to estimate and complete the missing fixation point based on the distribution of fixation points around the defective area. Analyze the relationship between the saccade data and the defective fixation point area. If the saccade passes through the defective area, correct the saccade trajectory based on the start and end points of the saccade and the normal eye movement pattern to ensure that the saccade trajectory conforms to normal visual behavior.

[0092] For the correction process of the scanning speed, glaucoma can be divided into mild, moderate and severe according to the severity of the disease. The scanning speed change pattern of patients with different degrees of glaucoma is obtained through clinical research. According to the severity of the subject's disease and the scanning speed change pattern, the scanning speed in the initial eye movement feature data is adjusted accordingly to improve the eye movement feature data.

[0093] For subjects with cataracts, it is necessary to adjust the eye movement speed in the initial eye movement feature data and remove abnormal gaze point data; the rotation frequency data is adjusted based on the average blinking frequency of the subjects to obtain eye movement feature data.

[0094] In this embodiment, adjusting the eye movement speed and removing abnormal fixation points can be achieved in the following ways: The eye movement speed adjustment ratio is set according to the severity of the cataract. Based on the adjustment ratio and the severity of the disease, the eye movement speed in the initial eye movement feature data is adjusted accordingly. A method similar to astigmatism is used to set the threshold by calculating the standard deviation of the fixation point position, remove abnormal fixation point data, and improve the quality of the eye movement feature data.

[0095] Adjusting the blink rate data can be achieved in the following ways: Calculate the average blink frequency of the subject over a period of time. According to the severity of the cataract, adjust the average blink frequency data in the initial eye movement feature data accordingly to obtain the final eye movement feature data.

[0096] In summary, adjusting data through eye diseases can improve the accuracy of eye movement data and enhance interaction accuracy. Eye control commands determined based on accurate data are more in line with the patient's intentions and avoid misjudgments in human-computer interaction.

[0097] In an optional embodiment, determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data includes: If the brain activity state information is normal, the correlation priority between each feature data in the user feature data and the eye control command is determined.

[0098] The eye control instructions of the subject are determined according to the relevance priority of each feature data and the eye movement feature data.

[0099] In this embodiment, since different eye movement feature data are set for different users, more than one eye control instruction may be matched when determining the eye control instruction. In order to avoid execution errors, the correlation priority between each feature data in the user feature data and the eye control instruction can be determined, and the eye control instruction to be executed first among the multiple eye control instructions can be determined according to the determined correlation priority of each feature data.

[0100] In an optional embodiment, determining the eye control instruction of the subject according to the correlation priority of each feature data and the eye movement feature data includes: Each feature data and eye movement feature data are matched with the rules in the preset rule library to obtain the target eye control instruction.

[0101] The subject's eye control instructions are determined according to the relevance priority of each feature data.

[0102] In this embodiment, the preset rule base is a database storing the relationship between various feature data combinations and corresponding eye control instructions. According to different feature data combinations, corresponding eye control instructions are defined. For example, for a person aged 20-30 years old, with brown pupil color, no eye disease, and eye movement characteristics of rapid scanning, the corresponding eye control instruction is defined as "switch page".

[0103] The various user feature data and eye movement feature data of the subject are input into the rules in the preset rule library to find the corresponding target eye control instructions. If there are multiple target eye control instructions, the target eye control instruction corresponding to the user feature data with the highest priority is used as the eye control instruction of the subject according to the priority of each user feature data; or, if there is only one target eye control instruction determined according to the eye movement feature data, but this instruction conflicts with the instruction matched according to the type of eye disease, the instruction matched by the eye movement feature data shall prevail.

[0104] In an optional embodiment, determining the correlation priority between each feature data in the user feature data and the eye control instruction includes: Analyze the correlation between various features in the user feature data and the eye control commands.

[0105] According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0106] In this embodiment, a correlation coefficient method may be used to analyze the correlation between each feature in the user feature data and the eye control command, so as to obtain the correlation priority between each feature data of different users and the eye control command.

[0107] In an optional embodiment, determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data includes: If the brain activity state information is: the first abnormal situation; then the area that the subject may currently focus on is predicted based on the subject's historical interaction data, and the area where the gaze point is located is updated based on the predicted result; and the two characteristic data of adaptation time and fixation time when different targets are switched are respectively subtracted from the preset error tolerance time to obtain the corrected adaptation time and updated fixation time; thereby obtaining new eye movement characteristic data; wherein the error tolerance time can be determined based on the adaptation time and fixation time of the subject and the adaptation time and fixation time of a normal person; for example, if the subject's adaptation time is 1.2 seconds and the normal person's adaptation time is less than 500 milliseconds, then the error tolerance time can be set to 700 milliseconds; If the brain activity state information is: the second abnormal situation; then the gaze point movement trajectory of the subject is obtained, the gaze point movement trajectory is filtered based on the adaptive Kalman filter or the sliding average filter method, the high-frequency chattering noise is dynamically filtered, and the real gaze point movement trajectory is obtained, and the eye movement feature data is corrected according to the real gaze point movement trajectory to obtain new eye movement feature data; The eye control instructions of the subject are determined according to the new eye movement feature data and the user feature data.

[0108] It should be noted that the method of determining the subject's eye control command based on the new eye movement feature data and the user feature data in this embodiment is the same as the method in the above embodiment, and will not be repeated here.

[0109] In this embodiment, if the brain activity state information indicates that different subjects may have different abnormal brain activity states (such as inattention, frequent switching of gaze points, slow reaction speed, etc.), corrections are made based on the eye movement characteristics affected by different abnormal conditions to obtain new eye movement characteristics. The eye control instructions generated based on the new eye movement feature data and user feature data compensate for the deviation of the eye movement characteristics caused by the abnormal brain activity state, so that the eye control instructions finally generated can accurately reflect the true intentions of the subjects and improve the accuracy of eye control interaction.

[0110] In summary, the embodiments of the present invention take into account the characteristics of different types of users, collect multiple user feature data and extract features in combination with eye movement data to obtain eye movement feature data that can accurately reflect the user's true eye movement intentions. These data can avoid misjudgment of commands due to ignoring individual differences, greatly improve the accuracy of determining eye control commands, and make the interaction more accurate. When determining eye control commands, the characteristics of different users are considered again so that the obtained eye control commands are adapted to different users, which improves the accuracy of eye control interaction and brings users a smoother and more natural interaction experience.

[0111] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0112] Figure 4 It is a structural schematic diagram of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention. like Figure 4 The eye-tracking-based eye-control interaction device provided in this embodiment includes: The data collection module 401 is used to collect the eye movement data and user characteristic data of the subject; wherein the user characteristic data includes age, pupil color and eye disease type; A feature extraction module 402, for extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data; The information generation module 403 is used to generate the brain activity state information of the subject according to the eye movement feature data; An instruction generation module 404 is used to determine the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data; The eye control interaction module 405 is used to perform eye control interaction based on the eye control instruction.

[0113] In a possible implementation, the feature extraction module 402 is specifically configured to determine pupil light reflectivity according to the pupil color of the subject; enhancing the pupil boundary in the eye movement data according to the pupil light reflectivity; Determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data; Determine the impact data based on the type of eye disease in question; Based on the impact data and the initial eye movement feature data, eye movement feature data is obtained In a possible implementation, the feature extraction module 402 is further configured to use the saccade amplitude and frequency as extraction indicators if the age of the subject is within a first preset age range; If the age of the subject is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators; If the age of the subject is within the third preset age range, the blinking frequency, the adaptation time when switching gazes between targets at different distances, the stability of the gaze point, the scanning speed and the gaze time are used as extraction indicators.

[0114] In a possible implementation, the feature extraction module 402 is further configured to: if the eye disease type is myopia, the influencing data is the saccade amplitude and frequency and the adaptation time when switching gazes to targets at different distances; If the eye disease type is hyperopia, the influencing data are the saccade amplitude, frequency, and fixation time; If the eye disease type is astigmatism, the influencing data are the saccade amplitude, frequency and fixation point stability; If the eye disease type is glaucoma, the influencing data are the saccade amplitude, frequency, scanning speed and the area where the fixation point is located; If the eye disease type is cataract, the influencing data are eye movement speed, gaze point stability, and blinking frequency.

[0115] In a possible implementation, the feature extraction module 402 is further configured to, if the eye disease type is myopia, remove abnormal eye movement data in the initial eye movement feature data based on the historical eye movement data of the subject and the predicted value of the eye movement data at the next moment, and adjust the adaptation time when switching gazes between targets at different distances according to the degree of myopia, so as to obtain eye movement feature data; If the eye disease type is hyperopia, based on the average mixed eye movement data of the subject, the saccade amplitude in the initial eye movement feature data is amplified, and the saccade frequency is reduced, and the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data; If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, the abnormal gaze point data in the initial eye movement feature data is removed, and the focus of the eye saccade is corrected based on the astigmatism degree to obtain the eye movement feature data; If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the eye saccade trajectory is corrected based on the relationship between the eye saccade data and the missing fixation point area, and the scanning speed is adjusted based on the severity of the disease to obtain the eye movement feature data; If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the blinking frequency data is adjusted based on the average blinking frequency of the subject to obtain the eye movement feature data.

[0116] In a possible implementation, the instruction generation module 404 is specifically configured to: if the brain activity state information is normal, determine the correlation priority between each feature data in the user feature data and the eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data and the eye movement feature data.

[0117] In a possible implementation, the instruction generation module 404 is specifically used to: match the various feature data and the eye movement feature data with the rules in the preset rule library to obtain the target eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data.

[0118] In a possible implementation, the instruction generation module 404 is further configured to: Analyze the correlation between various features in the user feature data and the eye control commands; According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

[0119] Figure 5 It is a structural schematic diagram of an eye-controlled interaction device based on eye tracking provided by an embodiment of the present invention.

[0120] like Figure 5 As shown, the eye-controlled interaction device 102 includes: a memory 1021 and a processor 1022. The memory 1021 stores a computer program, and the processor 1022 implements the steps of the above-mentioned method embodiments when executing the computer program. Alternatively, the processor 1022 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program.

[0121] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 1021 and executed by the processor 1022 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device 102.

[0122] The eye-controlled interaction device 102 may include, but is not limited to, a processor 1022 and a memory 1021. Those skilled in the art will appreciate that Figure 5 It is only an example of the eye-controlled interaction device 102 and does not constitute a limitation of the eye-controlled interaction device 102. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the eye-controlled interaction device 102 may also include input and output devices, network access devices, buses, etc.

[0123] The processor 1022 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0124] The memory 1021 may be an internal storage unit of the eye-controlled interaction device 102, such as a hard disk or memory of the eye-controlled interaction device 102. The memory 1021 may also be an external storage device of the eye-controlled interaction device 102, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the eye-controlled interaction device 102. Further, the memory 1021 may also include both an internal storage unit of the eye-controlled interaction device 102 and an external storage device. The memory 1021 is used to store computer programs and other programs and data required by the eye-controlled interaction device 102. The memory 1021 may also be used to temporarily store data that has been output or is to be output.

[0125] For the convenience and simplicity of description, only the division of the above functional modules / units is used as an example for illustration. In actual applications, the above functions can be assigned to different functional modules / units as needed. The above modules / units can be implemented in the form of hardware, software, or a combination of hardware and software.

[0126] The embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the methods in the above method embodiments are implemented.

[0127] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the methods in the above method embodiments are implemented.

[0128] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. If there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form a new embodiment according to their internal logical relationship.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An eye-controlled interaction method based on eye tracking, characterized in that: include: Collecting eye movement data and user characteristic data of the subject; wherein the user characteristic data includes age, pupil color and type of eye disease; Extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data; generating brain activity state information of the subject according to the eye movement feature data; Determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data; Based on the eye control instruction, eye control interaction is performed.

2. The eye-tracking-based eye-control interaction method according to claim 1, characterized in that: The step of extracting eye movement feature data of the eye movement data according to the eye movement data and the user feature data comprises: Determining pupil light reflectivity according to the subject's pupil color; enhancing the pupil boundary in the eye movement data according to the pupil light reflectivity; Determining an extraction index according to the age of the subject; and performing feature extraction on the eye movement data based on the extraction index to obtain initial eye movement feature data; Determine the impact data based on the type of eye disease in question; Eye movement feature data is obtained based on the impact data and the initial eye movement feature data.

3. The eye-controlled interaction method based on eye tracking according to claim 2, characterized in that: The step of determining the extraction index according to the age of the subject comprises: If the age of the subject is within the first preset age range, the saccade amplitude and frequency are used as extraction indicators; If the age of the subject is within the second preset age range, the saccade amplitude, frequency, fixation time, scanning speed and visual search efficiency are used as extraction indicators; If the age of the subject is within the third preset age range, the blinking frequency, the adaptation time when switching gazes between targets at different distances, the stability of the gaze point, the scanning speed and the gaze time are used as extraction indicators.

4. The eye-tracking-based eye-control interaction method according to claim 2, characterized in that: Determining the impact data according to the type of eye disease includes: If the eye disease type is myopia, the influencing data are the amplitude and frequency of eye saccades and the adaptation time when switching gazes to targets at different distances; If the eye disease type is hyperopia, the influencing data are the saccade amplitude, frequency, and fixation time; If the eye disease type is astigmatism, the influencing data are the saccade amplitude, frequency and fixation point stability; If the eye disease type is glaucoma, the influencing data are the saccade amplitude, frequency, scanning speed and the area where the fixation point is located; If the eye disease type is cataract, the influencing data are eye movement speed, gaze point stability, and blinking frequency.

5. The eye-control interaction method based on eye tracking according to claim 4, characterized in that: The step of obtaining eye movement feature data based on the impact data and the initial eye movement feature data includes: If the eye disease type is myopia, based on the historical eye movement data of the subject and the predicted value of the eye movement data at the next moment, the abnormal eye movement data in the initial eye movement feature data is removed, and at the same time, the adaptation time when switching gazes to targets at different distances is adjusted according to the degree of myopia to obtain the eye movement feature data; If the eye disease type is hyperopia, based on the average mixed eye movement data of the subject, the saccade amplitude in the initial eye movement feature data is amplified, and the saccade frequency is reduced, and the fixation time is corrected based on the degree of hyperopia to obtain the eye movement feature data; If the eye disease type is astigmatism, then based on the direction and angle of the subject's astigmatism axis, the abnormal gaze point data in the initial eye movement feature data is removed, and the focus of the eye saccade is corrected based on the astigmatism degree to obtain the eye movement feature data; If the eye disease type is glaucoma, the missing fixation point area in the initial eye movement feature data is completed, and the eye saccade trajectory is corrected based on the relationship between the eye saccade data and the missing fixation point area, and the scanning speed is adjusted based on the severity of the disease to obtain the eye movement feature data; If the eye disease type is cataract, the eye movement speed in the initial eye movement feature data is adjusted according to the severity of the disease, and the abnormal gaze point data is removed; the blinking frequency data is adjusted based on the average blinking frequency of the subject to obtain the eye movement feature data.

6. The eye-controlled interaction method based on eye tracking according to claim 1, characterized in that: The determining the eye control instruction of the subject according to the brain activity state information, the eye movement feature data and the user feature data comprises: If the brain activity state information is normal, determining the correlation priority between each feature data in the user feature data and the eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data and the eye movement feature data.

7. The eye-tracking-based eye-control interaction method according to claim 6, characterized in that: The step of determining the eye control instruction of the subject according to the correlation priority of each feature data and the eye movement feature data includes: Matching the various feature data and the eye movement feature data with the rules in a preset rule library to obtain a target eye control instruction; The eye control instruction of the subject is determined according to the relevance priority of each feature data.

8. The eye-tracking-based eye-control interaction method according to claim 6, characterized in that: The determining of the correlation priority between each feature data in the user feature data and the eye control instruction includes: Analyze the correlation between various features in the user feature data and the eye control commands; According to the results of the correlation analysis, each feature data is sorted to determine the correlation priority between each feature data and the eye control command.

9. An eye-controlled interactive device based on eye tracking, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. An eye-tracking-based eye-controlled interaction system, characterized in that: An eye tracker and an eye-controlled interaction device based on eye tracking as claimed in claim 9.

Images