Generation method for automatically generating eyeball movement data set, which is used for training artificial intelligence to classify digital eyeball movement videos
By generating an eye movement model to automatically generate annotated datasets, the problems of data shortage and high-cost annotation are solved, low-cost automated eye movement video classification is achieved, and the recognition accuracy of artificial intelligence is improved.
Patent Information
- Application Number
- CN202480011829.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-02-08
- Filing Date
- 2024-02-08
- Publication Date
- 2025-09-16
AI Technical Summary
In existing technologies, automated evaluation and artificial intelligence training of eye movement videos are limited by lack of data and high cost of manual labeling, making it impossible to achieve digital automated classification.
By generating an eye movement model, simulating the biological associations of input parameters, and setting impact parameters related to medical defects, a large number of annotated eye movement data sets are automatically generated and stored for training artificial intelligence for classification.
Low-cost, automated eye movement video classification has been achieved, improving the accuracy and efficiency of artificial intelligence in identifying medical defects.
Smart Images
Figure CN120660148A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for automatically generating an eye movement dataset for training artificial intelligence to classify digital eye movement videos, and a training method for training artificial intelligence using the eye movement dataset generated in this way. Background Art
[0002] Eye movements are known to indicate medical deficits. Therefore, specific eye movement tests can be used to identify different deficits in a patient's perception, particularly those related to the biological connections of the patient's cerebrum, cerebellum, inner ear, and brainstem regions. For example, some physicians are known to move a finger or other object in front of a patient and observe the patient's eye movements as they follow the finger, which can then be assessed. Depending on the medical deficit, a patient's eye movements may deviate significantly from normal, expected eye movements. For example, tracking speed, jumping eye movements, or problems with focusing may indicate different medical deficits.
[0003] A drawback is that this type of eye movement assessment, particularly the implementation and analysis of these eye movement tests, requires a high level of expertise and experience. Therefore, digital assessment, particularly automated evaluation of the digital videos collected for these eye movement tests, is currently only feasible to a limited extent. Furthermore, the application of AI in this field is often limited by the lack of sufficient classified data to train it. This is because thousands of eye movement videos must be collected and provided to experts for classification to generate training data for this AI classification task, which is a tremendous workload. Furthermore, it is necessary to ensure that these manually annotated videos contain a sufficient proportion of significant eye movement features associated with medical defects. Therefore, the training and basic application of AI are limited not only by the lack of such data but also by the extensive human effort and high cost required to manually annotate and classify this data. Consequently, automated evaluation of digital eye movement videos is currently not possible. Summary of the Invention
[0004] The present invention aims to at least partially solve the above-mentioned problems. In particular, the present invention aims to automate the task of classifying digital eye movement videos in a low-cost and simple manner.
[0005] The above-mentioned object is achieved by the generation method described in claim 1 and the training method described in claim 13. Further features and details of the present invention are set forth in the dependent claims, the description, and the accompanying drawings. Features and details relating to the generation method of the present invention also apply to the training method of the present invention, and vice versa. Therefore, cross-references may be made when describing the various key aspects of the present invention.
[0006] According to the present invention, a method for automatically generating an eye movement dataset is proposed for training artificial intelligence to classify digital eye movement videos. The method comprises the following steps:
[0007] - providing an eye movement model that simulates the biological correlation between at least one input parameter and the eye movement induced thereby, and taking into account at least one influencing parameter,
[0008] - setting at least one influencing parameter related to a certain medical defect for the provided eye movement model,
[0009] - setting at least one input parameter related to a certain eye movement test for the provided eye movement model,
[0010] - Generate an eye movement dataset using the eye movement model according to the set influence parameters and input parameters,
[0011] - storing the generated eye movement datasets in an eye movement database along with the medical defects associated with the influencing parameters as labels,
[0012] - Repeating the steps of inputting, generating and storing, and changing at least one influencing parameter and / or at least one input parameter.
[0013] The core concept of this invention is to generate a dataset that can be used to train an AI to learn and master the classification capabilities of specific medical defects based on digital eye movement videos. In other words, the method aims to enable AI to classify patients into a specific medical specialty based on digital eye movement videos, which will be responsible for treating the disease or problem related to the identified classification result (i.e., medical defect). For example, this method can be used to triage patients to otolaryngology, neurology, or other specialized departments based on the classification results, so that more accurate and specific medical analysis and corresponding medical diagnosis plans can be developed.
[0014] Unlike previous, costly manual data generation methods, the present invention's generation method should enable at least a partially, and preferably fully, digital and automated process. The core of this generation method is an eye movement model that algorithmically simulates the biological relationship between external input parameters and internal (i.e., patient-related) influencing parameters. In other words, the eye movement model is a simplified algorithmic modeling of the functional processes of the patient's brain, muscles, and nervous system.
[0015] Input parameters are external parameters, particularly those related to eye movement testing. For example, here's a description of a motor test commonly performed by medical professionals. A common eye movement test involves a professional making a specific left-right finger movement in front of a patient, guiding them to follow the finger or object with their eyes. Medical deficiencies in the patient's eye muscles, perception, or nervous system can affect their ability to follow an object with their eyes.
[0016] The eye movement model is able to calculate the corresponding eye movements based on input parameters (such as the trajectory and movement coordinates of an object) in combination with influencing parameters (i.e., the patient's internal biological connections set in the algorithm). Since this involves relatively simple biological connections, it can also be expressed using simplified algorithmic connections. In addition, various changes can be implemented in these algorithmic connections, in particular, the above-mentioned influencing parameters can be changed, so that medical defects can be simulated algorithmically in the eye movement model. For example, by setting different influencing parameters, the signal transmission speed, transmission quality or performance of the patient's eye muscles can be adjusted in a qualitative and / or quantitative manner in the model.
[0017] Healthy patients can convert received input parameter signals into corresponding eye movements through their eye muscles in near real time. However, patients with medical eye muscle defects can recognize object movement, but the conversion into normal eye movements may be delayed or even impossible. In this case, by adjusting the influencing parameters, this delay will manifest as different eye movement data in the model. In other words, the eye movement model can use external input parameters from the patient's perspective, combined with its biological simulation algorithm, to generate an eye movement dataset. By introducing different influencing parameters into the algorithmic structure of this eye movement model, the algorithm can simulate different medical defects and even quantitatively adjust the degree of the defect. Therefore, this eye movement model can simply output movement information in the form of eye movement coordinates or directions, thereby forming an eye movement dataset.
[0018] The generation method proposed in this invention is based on this model and operates under the premise of setting at least one influencing parameter associated with a medical defect. In other words, by selecting a certain influencing parameter, the corresponding medical defect can be selected in the eye movement model and set in the eye movement model, thereby adjusting the eye movement model for that specific medical defect and generating eye movement data related to it. In addition, at least one input parameter, such as the left-right movement of the object mentioned above, must be set to represent a specific eye movement test. Based on these two parameters (input parameter and influencing parameter), the eye movement model is used to generate a corresponding eye movement dataset. This dataset can then be stored in an eye movement database. During storage, the generated eye movement dataset can be saved along with the medical defect label associated with the set influencing parameter. From the perspective of the training method described below, these are digitally generated datasets that are considered to have been labeled or classified during the training process due to their association with a specific medical defect. In other words, the generated eye movement dataset is stored specifically and in relation to the medical defect previously associated with the influencing parameter.
[0019] In the last step of the generation method described in the present invention, the steps of setting, generating and storing are repeated many times, in particular frequently, and the influencing parameters and / or input parameters are at least partially changed. These changes can be large or small. Large changes refer to switching different medical defects at the influencing parameter level, or using different eye movement test methods at the input parameter level. In addition, it is also conceivable that, based on switching between different medical defects and / or different eye movement tests, quantitative change options within a medical defect and / or within a certain eye movement test are provided separately or additionally. Through the combination of the above-mentioned large and small changes, a large number of parameter combinations and variations can be generated under the premise of low complexity, thereby automatically generating a large number of eye movement data sets that are specific, different and correctly labeled.
[0020] In other words, this makes it possible to build a database that can be automatically generated and populated with a large number of automatically correctly labeled eye movement datasets. These classified eye movement datasets are very large in number and can be used in subsequent training methods to train artificial intelligence to automatically or at least semi-automatically classify digital eye movement videos in practical applications through their correspondence with medical defects.
[0021] To illustrate the above abstract process more specifically, a simple example is provided below. For example, in the eye movement model, it can be represented as a simple algorithmic association that describes the process by which a patient recognizes image information through their eyes. For example, an eye movement test can use the repeated left and right movement of an object as the test action. In this case, the input parameters are the motion coordinates of the object's trajectory during the eye movement test. Other input parameters may also include the camera's shooting parameters and the patient's head movement, which will be explained in detail later. In the process of digitally performing eye movement testing, the eye movement model can apply the implemented biological algorithmic association to the input parameters, thereby generating eye movement data based on the test input parameters and the biological association.
[0022] When the influencing parameters are set to correlate with a specific medical defect, the generated eye movements will be consistent with that defect. In other words, the influencing parameters can algorithmically adjust the biological signal transmission pathways in the model, such as reducing or completely blocking their transmission pathways. In extreme cases, if neural signal transmission is severely affected, such as if the recognized object movement cannot be transmitted to the eye muscles in the eye movement model, then the tracking movement may be very slow. Therefore, by adjusting the influencing parameters accordingly, not only can the qualitative switching of medical defects be achieved, but also quantitative adjustment can be achieved.
[0023] In short, not only can the data generation process itself be digitized and automated, but the parameter variation process can also be brought under the scope of automated control. This means that, for example, using a pre-set eye movement model, the same amount of eye movement datasets can be automatically generated through thousands of automatic variations. These datasets can then be stored as labeled data and directly used in subsequent training methods.
[0024] If the eye movement data set is grouped and saved according to preset influencing parameters in the data generation method of the present invention, more advantages will be brought. Grouping the eye movement data set by influencing parameters can also be understood as grouping the data set by medical defects. This grouping method makes it possible to construct the relevant eye movement database more finely, so that each part of it can be dedicated to the training of artificial intelligence. For example, it can be imagined that for different medical defects, artificial intelligence can use the corresponding parts of the database to specifically train specific medical eye movement test models. Therefore, in subsequent eye movement video classification applications, different artificial intelligence modules specially trained for specific medical defects can be used. Compared with general artificial intelligence modules, this specifically trained artificial intelligence can perform more professional and targeted training, thereby obtaining more accurate classification results with lower error rates.
[0025] In addition, if at least one of the following is set as an input parameter in the generation method of the present invention, better results will be achieved:
[0026] - The motion coordinates of the camera that records eye movements,
[0027] - The motion coordinates of the patient's head including the eyeballs,
[0028] - The motion coordinates of the object used to perform the eye movement test.
[0029] The above list is not exhaustive. Of course, two or more of these input parameters can be combined or used simultaneously. These motion coordinates and their settings are also preferably purely virtual and digital, processed as part of the generation method. Specifically, these motion coordinates constitute auxiliary information used as input parameters, providing secondary information related to the eye movement test or its current environment. For example, in future scenarios for eye movement video classification and artificial intelligence applications, a mobile phone camera can be used to film the patient, particularly their head. During the eye movement test, a medical professional can hold the phone in one hand while using the other to manipulate the object guiding the eye movement test. When the patient focuses on the object, at least three independent movements typically occur, which are reflected in the 2D or 3D image captured by the phone. First, there is the movement of the object itself, which is the primary movement in the eye movement test. Second, because the camera is handheld by the operator, it is not completely still and also experiences its own movement, introducing additional variation. However, this motion variation can be avoided by mounting the camera on a tripod. Finally, the patient's head is not completely still, so it may also produce small movements, which will also be converted into motion coordinates, external input factors, and ultimately become input parameters of the eye movement model.
[0030] If, in the generation method of the present invention, at least one influencing parameter is varied in a quantitative manner, consistency with a specific medical defect can be ensured. As previously described, one or more medical defects can be activated or deactivated via the influencing parameter. By varying the influencing parameter, even varying degrees of the medical defect can be simulated within the algorithmic eye movement model. Of course, different medical defects can also be combined, further increasing the potential for variation and significantly increasing the number of specific and high-quality training datasets, i.e., eye movement datasets.
[0031] Another advantage of the present invention is that the eye movement data can be rendered during the generation process, either before or after the dataset is saved. Rendering refers to the digital generation and rendering of video data, in addition to simple eye center coordinates such as the pupil center. This video data can then be further processed, loaded onto a digitized face or head, or further processed through filters described below to provide a rendered and automatically generated eye movement dataset that can be directly compared with real-world eye movement videos that are then required for classification.
[0032] Advantages are also realized if the setting and changing of at least one input parameter in the generation method of the present invention is continuous. As previously explained, the input parameters can be constructed in the form of motion coordinates. The continuity of these motion coordinates is a realistic prerequisite. While there may be broken lines, changes, or jumps, there must be no "breaks" where the object, eye, or camera instantly jumps from one position to another, resulting in a movement path that has not been truly experienced. This continuity allows for more realistic generation of eye movement datasets for training.
[0033] In the data generation method of the present invention, it is also advantageous to set degrees of freedom for changes in at least one influencing parameter and / or input parameter. For example, a random number generator can be used to achieve automatic changes in parameters, thereby further improving the degree of automation of the dataset generation process. In order to ensure that the generated eye movement dataset is still reasonable and realistic even under conditions of automatic changes, a certain range of degrees of freedom can be set according to the real scene. For example, for eye movement tests, a common range of movement of the object can be set, a range that must not be exceeded even during the automatic change process. Similarly, a range of movement can also be set for the camera or the patient's head, which also corresponds to the common action boundaries in eye movement tests.
[0034] In addition, the generation method of the present invention can be further optimized: the generated eye movement dataset is filtered or enhanced before or after saving, especially by adding data noise. This filtering process can provide an eye movement dataset that is closer to reality and more realistic. Thereby, the generated automated eye movement dataset is more realistic and can be better compared with the real eye movement videos used in subsequent artificial intelligence classification tasks. In other words, the filtering process deliberately "reduces" the quality of the automatically generated eye movement dataset to enhance the realism and practicality of this data. For the application scenarios of artificial intelligence training, the higher the realism of the training dataset, the higher the accuracy of the trained artificial intelligence module in subsequent classification tasks.
[0035] The generation method of the present invention has the further advantage that the eye movement model used comprises at least one of the following modeled influence components:
[0036] - Eye muscles,
[0037] - Signaling pathway to eye muscles,
[0038] - Cerebellum,
[0039] - brainstem,
[0040] - semicircular canals,
[0041] -cerebral cortex.
[0042] The above list is not exhaustive. Preferably, these influencing components are simulation models of biological influencing components in the patient's brain or muscles. Each of these influencing components can be implemented in the algorithm as a submodule in the eye movement model, so each submodule can be assigned one or more corresponding influencing parameters. In other words, a specific medical defect (such as a biological influencing component mentioned above) can be selected and the specific manifestation of the defect can be simulated by adjusting the relevant influencing parameters. For example, a decrease in the responsiveness or execution ability of the eye muscles can represent a certain medical defect. The signal pathway to the eye muscles may also be damaged due to the medical defect. This defect can be a qualitative or quantitative manifestation. Similarly, different areas of the brain, such as the cerebellum, brainstem or cerebral cortex, can also be used as influencing components and changed by setting the influencing parameters to simulate the corresponding neurological medical defects in the process of generating the eye movement data set.
[0043] In addition, the generation method of the present invention also has an advantage, that is, when setting the influencing parameters, selection can be made based on a predefined set of medical defects and their corresponding input parameter sets. Although a large number of data sets can be generated by automatically changing the input parameters and / or influencing parameters, specific medical defects can also be simulated through preset influencing parameter combination patterns or input parameter combination patterns. In this way, the real-world relevance of the generated data set will be further improved, thereby improving the performance of the trained artificial intelligence module in eye movement video classification. Various combinations of input parameters or influencing parameters can form a pattern library of medical defect patterns and / or eye movement test patterns.
[0044] Another advantage is that, in the generation method of the present invention, the eye movement model uses at least one of the following databases when generating the dataset:
[0045] - head movements,
[0046] -Camera movement,
[0047] - object movement,
[0048] - human face,
[0049] -background,
[0050] -Camera model.
[0051] The above list is not exhaustive. In addition to considering different input parameters, the eye movement dataset output by the eye movement model may also include different representations. This processing is particularly suitable when it is desired to provide eye movement renderings from the eye movement dataset, thereby providing direct training of artificial intelligence based on the rendered digital video data.
[0052] A further advantage of the method of the present invention is that the eye movement model preferably includes at least one random generator, which is particularly used to randomly vary at least one influencing parameter. In addition or as an alternative, it is also feasible to use a random generator to vary at least one input parameter. Preferably, these random generators should be provided with limiting numerical boundaries, which can be defined in the form of the aforementioned degrees of freedom. The use of a random generator is a very simple and low-cost method that can automate the steps of parameter variation so that the entire generation process can in principle be fully automated after startup, and can generate thousands or even more than ten thousand labeled eye movement data sets and store them in the corresponding eye movement database.
[0053] Another aspect of the present invention is to use the eye movement dataset generated by the present invention to train artificial intelligence for classification of digitized eye movement videos. Therefore, the training method proposed in the present invention has the same advantages as the generation method based on the present invention described in detail above. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Other advantages, features and details of the present invention will be further clarified by the following description, in which embodiments of the invention will be described in detail with reference to the accompanying drawings. The accompanying drawings are schematically shown as follows:
[0055] Figure 1 One embodiment of the generation method of the present invention,
[0056] Figure 2 Another embodiment of the generation method of the present invention,
[0057] Figure 3 An eye movement test implementation method related to the present invention. DETAILED DESCRIPTION
[0058] Figure 1 The implementation process of the generation method is schematically shown. The generation device 10 can be a computer device that implements the various algorithms on a computer chip and makes them executable. The core of the generation method and the core of the generation device 10 is an algorithm-based eye movement model 20, which includes two influencing components and is based on two variable influencing parameters 24. When performing eye movement testing, it can be as follows Figure 3 Proceed as shown.
[0059] Figure 3The diagram illustrates a situation in which a patient 220 has two eyes 222 on their head. During an eye movement test, a medical professional will move an object (numbered 230) multiple times from left to right along a trajectory with motion coordinates BK. The patient is instructed to follow the movement of object 230 with their eyes 222. The following movement of eyes 222 constitutes the eye movement behavior to be generated by the eye movement model 20, i.e., the eye movement video as the test result. During the movement along motion coordinates BK, the perceptual connections between the patient's neural pathways and muscles come into play, prompting eyes 222 to follow the movement.
[0060] Figure 1 This biological association is schematically illustrated in the eye movement model 20, and when the medical defect MD is set, the corresponding two influencing parameters 24 can be qualitatively and / or quantitatively adjusted. The input parameter 22 can be the motion coordinates BK of the object 230. The input parameter 22, combined with the influencing factor 24 of the medical defect MD, can be used to artificially simulate the eye movement behavior of the eye 222 during the test, i.e., digitized eye movement data. Such movement trajectories can preferably be generated in the form of motion coordinates of the pupil, iris center, or eye orientation, or can be actual digitized video, output as an eye movement dataset 30, and stored in the eye movement database 40 along with the specified medical defect MD. After completing a generation process, the operation will be repeated, during which at least one parameter change will be made. This can be a qualitative and / or quantitative change to one or more input parameters 22 and / or one or more influencing parameters 24.
[0061] Figure 2 This process of parameter changes and multiple executions is shown, and in three runs, three different eye movement datasets 30 with the same or different medical defects MD are generated. If this process is executed a large number of times (especially thousands of times), the eye movement database 40 can be filled with thousands of annotated data. Figure 2 In the embodiment, the eye movement database 40 is further divided into detailed structures according to different medical defects MD. In this way, the eye movement dataset 30 corresponding to each specific medical defect MD can be accurately extracted and used for independent training methods.
[0062] Look at the reference again Figure 3, it can be seen that the input parameter 22 can have a wider range of variations. Although the eye movement test is basically set by the movement of the object 230 along the movement coordinate BK, other components of the test may also have an impact on the patient's eye movements. In this example, these influences include the movement of the camera 210 in the test system 200 and the movement of the head 220. The figure also shows how the captured eye movement video 110 is passed to the artificial intelligence 100 for classification in the case of the test system 200, and based on Figure 1 and Figure 2 The artificial intelligence 100 is trained using an eye movement dataset 30 from an eye movement database 40 .
[0063] The above description of the embodiments describes the present invention by way of example only.
[0064] Reference Signs List
[0065] 10 Generator
[0066] 20 Eye Movement Model
[0067] 22 Input Parameters
[0068] 24 Influencing Parameters
[0069] 30 Eye Movement Datasets
[0070] 32 Eye Movement Rendering
[0071] 40 Eye Movement Database
[0072] 100 Artificial Intelligence
[0073] 110 Eye Movement Video
[0074] 200 test system
[0075] 210 cameras
[0076] 220 Head
[0077] 222 Eyes
[0078] 230 objects
[0079] MD Medical Deficiency
[0080] BK motion coordinates
Claims
1. A method for automatically generating an eye movement dataset (30) for training an artificial intelligence (100) to classify digital eye movement videos (110), comprising the following steps: - providing an eye movement model (20) that simulates the biological correlation between at least one input parameter (22) and the eye movement induced thereby, and taking into account at least one influencing parameter (24), - setting at least one influencing parameter (24) associated with a certain medical defect (MD) for the provided eye movement model (20), - setting at least one input parameter (22) associated with a certain eye movement test for the provided eye movement model (20), - Based on the set influence parameters (24) and input parameters (22), use the eye movement model (20) Generate an eye movement dataset (30), - The generated eye movement dataset (30) is combined with the medical defects associated with the influencing parameters (24) (MD) as a label and stored together in the eye movement database (40), - Repeating the steps of inputting, generating and storing and changing at least one influencing parameter (24) and / or at least one input parameter (22).
2. The generation method according to claim 1, characterized in that When stored, the eye movement data sets (30) are grouped according to the set influence parameters (24).
3. The generation method according to any one of the preceding claims, characterized in that The input parameter (22) includes at least one of the following data: - the motion coordinates (BK) of the camera (210) that records eye movements, - motion coordinates (BK) of the patient's head (220) including the eyes (222), - Movement coordinates (BK) of the object (230) used for performing the eye movement test.
4. The generation method according to any one of the preceding claims, characterized in that The change in at least one influencing parameter (24) is at least partially a quantitative change to maintain consistency with a specific medical defect (MD).
5. The generation method according to any one of the preceding claims, characterized in that During and / or after storing, the eye movement dataset (30) includes an eye movement rendering (32).
6. The generation method according to any one of the preceding claims, characterized in that When at least one input parameter (22) is set and changed, the change process is continuous.
7. The generation method according to any one of the preceding claims, characterized in that When changing at least one influencing parameter (24) and / or input parameter (22), degrees of freedom are set.
8. The generation method according to any one of the preceding claims, characterized in that Before or after storage, the generated eye movement dataset (30) will be filtered or processed, in particular in order to add data noise to the eye movement dataset (30).
9. The method according to claim 1, wherein The eye movement model (20) has a modeling influence component that includes at least one of the following: - Eye muscles, - Signaling pathway to eye muscles, - Cerebellum, - brainstem, - semicircular canals, -cerebral cortex.
10. The method of generating according to claim 1, characterized in that When the influencing parameter (24) is varied, a selection is made from a predefined selection of medical defects (MDs) and corresponding input parameters (22).
11. The method according to claim 1, wherein The eye movement model (20) calls at least one of the following databases when generating the eye movement dataset (30): - head movements, -Camera movement, - object movement, - human face, -background, -Camera model.
12. The method according to claim 1, wherein The eye movement model (20) uses at least one random generator, in particular randomly varying at least one influencing parameter (24).
13. A training method for training an artificial intelligence (100) for classifying digital eye movement videos (110) using an eye movement dataset (30) generated by the feature generation method according to any one of claims 1 to 12.