An eye movement trajectory data synthesis method for people with visual field defects

Abstract

The application provides an eye movement track synthesis method for people with visual field defects, and relates to the fields of human-computer interaction and virtual simulation, in particular to eye movement behavior classification, eye movement track synthesis, eye movement data enhancement, and specifically relates to an eye movement track data synthesis method for people with visual field defects. The application is mainly applied to track synthesis and data enhancement for people with visual field defects, can synthesize eye movement tracks of people with visual field defects, and can greatly reduce the difficulty and complexity of eye movement data collection of people with visual field defects, and reduce time cost and economic cost. The application can simulate and synthesize eye movement track data of any visual field defect type, and provides great convenience for data acquisition of reading, driving and walking behavior research of people with visual field defects.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and virtual simulation technology, and more particularly to a method for synthesizing eye-tracking data for people with visual field defects. Background Technology

[0002] There are two main types of visual field defects: one is damage to the concentric visual area caused by conditions such as retinitis, choroidal loss, or glaucoma, also known as tubular visual field; the other is hemianopsia, mostly ipsilateral hemianopsia. By fixing a prism module near the upper and / or lower part of the residual central visual field, artificial visual aids are created in the peripheral vision, which can help expand the visual field of people with visual field defects. Compared with people with normal visual field, people with visual field defects have a smaller visible visual field, fewer visible fixation points, and altered visual behavior, with significant differences in saccade paths and fixation frequencies.

[0003] Depending on the area of ​​visual field loss, visual field defects are mainly classified into three types: central visual field defects, peripheral visual field defects, and partial visual field defects. Visual field expansion methods can visualize parts of scenes that were originally invisible within the visible visual field, which is beneficial to the daily lives of people with visual field defects. However, due to the change in the visual field, the eye movement trajectories of people with visual field defects differ significantly from those of normal individuals. Currently, most eye movement trajectory synthesis methods on the market are designed for normal individuals, lacking methods for synthesizing eye movement trajectories for people with visual field defects. Furthermore, research on the attention mechanisms of people with visual field defects requires the collection of large amounts of data from these individuals, typically incurring significant time and financial costs, and it is uncertain whether stable and usable eye movement trajectories can be obtained. Summary of the Invention

[0004] In response to the technical problems mentioned in the background section, this invention provides a method for synthesizing eye movement trajectory data for individuals with visual field defects. The invention generates synthesized eye movement trajectories based on partial real eye movement trajectories using a data augmentation module.

[0005] The technical means employed in this invention are as follows:

[0006] An eye-tracking synthesis method for individuals with visual field defects, used for eye-tracking synthesis and data augmentation in this population, includes the following steps:

[0007] Step 1: Setting the visual field area; Obtain the visual field area of ​​the visual field defect population through visual field measurement and visual field expansion rules, and complete the visual line plane calibration within the visible area using multiple generated reference calibration points;

[0008] Step 2: Data generation; acquire gaze data within the visual field, use continuous principal component clustering to distinguish gaze data into fixation data and saccade data, combine time information and landing area judgment to generate raw eye movement trajectory data;

[0009] Step 3: Adversarial learning; Train a gaze generation adversarial model and a saccade generation adversarial model based on the original eye movement trajectory information. Generate gaze data and saccade data through the two models respectively. Connect the generated data clusters to form eye movement trajectory data. Compare and learn the data in the actual scene to optimize it, and remove eye movement trajectories that do not conform to the visual field settings.

[0010] Furthermore, the setting of the field of view area includes the following steps:

[0011] Step 11: Use a perimeter to measure the user's horizontal and vertical field of vision angles, as well as the expected distortion or clipping area, to obtain field of vision measurement data;

[0012] Step 12: Convert the field of view measurement data into grayscale values ​​or grayscale levels in the field of view coordinate system;

[0013] Step 13: Measure the distance from the user to the screen, and use trigonometric functions to calculate the calibration mapping of the human eye's field of vision on the screen to obtain the approximate areas of the visible and invisible regions;

[0014] Step 14: Smooth the mapped region of the visible area using Gaussian filtering;

[0015] Step 15: Use morphological erosion to narrow the boundaries of the field of view;

[0016] Step 16: After obtaining the visible area, invisible area, and extended area, you can mark them on the screen;

[0017] Step 17: After marking the field of view areas, perform line of sight calibration; set the reference calibration point in the middle part of the visible field of view area. If there are multiple visible field of view areas, select the largest visible field of view area; if the visible field of view area is less than 5% of the total field of view area, use the center point of each visible field of view area as the reference calibration point.

[0018] Step 18: After selecting the reference calibration point, the user is asked to look at the reference calibration point in sequence; at the same time, the coordinates of the reference calibration point and the human eye characteristics are recorded during the looking process, and the eye-tracking method is used to complete the eye-tracking mapping between the human eye and the screen.

[0019] Step 19: Repeat step 18 until all angles are calibrated, and during the repetition, rotate the rectangle 45° clockwise each time.

[0020] Furthermore, the selection criteria for the reference calibration point are as follows: take the rectangle with the largest area in the visible field of view, and the four vertices and the midpoint of the rectangle are the reference calibration points.

[0021] Furthermore, the field of view measurement data is represented in the form of angles or positional information relative to the center of the field of view.

[0022] Furthermore, the data generation includes the following steps:

[0023] Step 21: After acquiring gaze data, classify and cluster eye movements; analyze the principal components of the clusters to obtain sub-clusters, until the lowest-level gaze point is obtained, thus obtaining all gaze behaviors;

[0024] Step 22: Perform corresponding processing according to the area where the landing point is located; the area where the landing point is located includes: the visible field of view area, the invisible area, the extended field of view area, and the overlapping field of view area;

[0025] For the visible visual field, eye movement behavior is divided into fixation behavior and saccade behavior;

[0026] No processing is performed on areas that are not visible to the naked eye;

[0027] For expanding the visual field, saccades are considered fixation behavior;

[0028] For overlapping visual fields, fixation behavior between the visible visual field and the extended visual field is considered as saccade behavior.

[0029] Step 23: Obtain eye movement behavior; the eye movement behavior includes: fixation behavior between visible areas, saccade behavior within the visible area, fixation behavior between the same extended areas, saccade behavior between different extended areas, and saccade behavior between the visible area and the extended area;

[0030] Step 24: After classifying the eye-tracking behavior, the original motion trajectory is synthesized using the time information recorded by the eye-tracking method.

[0031] Furthermore, the eye movement behavior includes fixation behavior and saccade behavior; the fixation behavior is manifested as a dense cluster of points, and the saccade behavior is the crossing from one cluster of points to another. The movement between different fixation behaviors is saccade behavior.

[0032] Furthermore, the adversarial learning includes the following steps:

[0033] Step 31: Input the original trajectory information into the gaze generation adversarial model and the saccade generation adversarial model respectively; both the gaze generation adversarial model and the saccade generation adversarial model contain a generator and a discriminator;

[0034] Step 32: The generator creates synthetic data based on the original gaze point data, which is used to train the discriminator;

[0035] Step 33: Synthesize gaze trajectory sequences through batch normalization and activation functions; each sequence starts from the first gaze position obtained by the Generative Adversarial Network (GAN) from simulated gaze behavior, and each subsequent gaze position is added to the current sequence. The position between two gaze clusters determines the amplitude of the saccade behavior.

[0036] Step 34: The discriminator uses the gaze trajectory sequence as input to distinguish whether the gaze trajectory sequence is a real sequence or a sequence generated by the generator;

[0037] Step 35: Perform comparative learning optimization and delete eye-tracking trajectories that do not conform to the regional settings. Compared with the prior art, the present invention has the following advantages:

[0038] This invention uses principal component analysis to classify eye-movement behavior and generative adversarial networks to augment gaze trajectories. The main steps of this invention all yield beneficial effects.

[0039] First, by using calibration points in the visible area to perform multiple line-of-sight calibrations on people with visual field defects, the correct mapping between eye movements and visual field areas was achieved, with overall high accuracy.

[0040] Second, by using continuous principal component clustering, gaze data is divided into fixation data and saccade data. Combined with time information and landing area judgment, the original eye movement trajectory data is generated accurately and stably.

[0041] Third, by using two generative adversarial networks and comparative learning to obtain accurate and robust synthetic gaze trajectories, the complexity of data collection is greatly reduced, as well as the time and economic costs incurred.

[0042] This invention can be used to simulate and synthesize eye movement trajectory data of any type of visual field defect, which is of positive significance for research on the behavior of people with disabilities in reading, driving, and other aspects. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is the overall flowchart of the present invention.

[0045] Figure 2This is a schematic diagram illustrating the field of view of the present invention.

[0046] Figure 3 This is a schematic diagram of eye-movement behavior clustering according to the present invention.

[0047] Figure 4 This is a schematic diagram of the eye movement behavior classification of the present invention.

[0048] Figure 5 Generate an adversarial model structure diagram for this invention.

[0049] Figure 6 This is a schematic diagram illustrating the comparative learning and optimization of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] like Figure 1-6 As shown, this invention provides an eye-tracking synthesis method for people with visual field defects, used for eye-tracking synthesis and data augmentation in people with visual field defects, including the following steps:

[0053] Step 1: Visual field area setting; The visual field area of ​​the person with visual field loss is obtained by visual field measurement and visual field expansion rules and marked on the screen. Reference calibration points are generated multiple times in the visible area, and the user completes the line of sight calibration by looking at the calibration points.

[0054] Includes the following steps:

[0055] A perimeter is typically used to measure the user's horizontal and vertical visual field angles, as well as any potential distortions or clipping areas. Visual field measurement data is usually represented as angles or positional information relative to the center of the visual field. After obtaining the measurement data, it needs to be converted into grayscale values ​​or grayscale quantization levels in the visual field coordinate system to express visual acuity or other relevant information. The distance from the user to the screen in front is measured, and a calibration mapping of the human eye's visual field on the screen is calculated using trigonometric functions to obtain the approximate areas of the visible and invisible regions. Gaussian filtering is then used to smooth the mapped area to reduce noise and detail, ensuring the stability and accuracy of subsequent processing. Finally, morphological erosion is used to narrow the boundaries of the visual field in the measurement results. This step helps to accurately define the boundaries of the visible and invisible regions, resulting in results with high confidence. The expansion of the visual field usually follows certain rules, thus yielding the expanded visual field area. Once the visible, invisible, and expanded areas are obtained, they can be labeled on the screen.

[0056] After marking the visual field area, gaze calibration is required. Reference calibration points will appear in the middle of the visible visual field area, typically five in sequence. The selection criteria for calibration points are to choose the largest rectangle within the visible visual field area; its four vertices and the midpoint of the rectangle constitute the reference calibration point. If multiple visible visual field areas exist, the largest one is selected to complete the above steps. If the overall visible visual field area is small (if it is less than 5% of the total visual field area, the center point of each visible visual field area is used as the reference calibration point), a single area's center point can be used. After selecting a reference calibration point, the user is instructed to gaze at it sequentially. During this process, the calibration point coordinates and the user's eye characteristics are recorded, and gaze mapping between the user's eyes and the screen can be achieved through gaze tracking. Considering the special characteristics of individuals with visual field defects, this step should be repeated 2-3 times until all angles are calibrated. Each time, the rectangle should be rotated 45° clockwise to reselect the largest rectangle and determine a new reference calibration point.

[0057] Step 2: Data Generation; Acquire gaze data within the visual field, and use continuous principal component clustering to distinguish gaze data into fixation data and saccade data. Combine time information and saccade location judgment to generate raw eye movement trajectory data. This includes the following steps:

[0058] After gaze calibration, the user is asked to look at the screen. The camera then collects relevant information, and gaze tracking methods are used to obtain gaze data on the screen, including gaze position and time information. After acquiring the gaze data, eye movements need to be categorized. The eye movements discussed here are mainly divided into fixation and saccades. Fixation is characterized by dense clusters of points, while saccades are characterized by moving from one cluster of points to another. For a gaze record, since the distribution of user fixation points varies in density (e.g., the coordinate density of the fixation area is higher than that of the non-fixation area), we can use the current principal components as clusters, then analyze the principal components of the clusters to obtain sub-clusters, until we obtain the lowest-level fixation points. Following this process, all fixation behaviors can be obtained. Movement between different fixation behaviors is called saccade.

[0059] After classifying eye movements, corresponding processing is needed based on the region where the gaze point is located. In addition to the invisible and visible regions, individuals with visual field defects also have extended regions that partially overlap with the visible region. Therefore, the gaze point region can be divided into four types: visible region, invisible region, extended region, and overlapping region, with data from the invisible region not considered.

[0060] For the visible visual field, eye movements are categorized into fixation and saccades. For the extended visual field, since the extended area is relatively small, the difference between saccades and fixation is minimal; therefore, saccades within the extended area are considered fixation, while saccades between two different extended areas remain unchanged. For overlapping areas, since the original positions may not be adjacent, fixation between the visible and extended visual fields is considered saccades. Based on the jump location of the landing area, eye movements can be classified into five categories: fixation between visible areas, saccades within the visible area, fixation between the same extended area, saccades between different extended areas, and saccades between the visible and extended areas. After classifying the eye movements, the original motion trajectory can be synthesized using time information recorded through eye-tracking methods.

[0061] Step 3: Adversarial Learning; Using raw eye movement trajectory information, train a gaze generation adversarial model and a saccade generation adversarial model. These two models generate gaze data and saccade data respectively. The generated data clusters are concatenated to form eye movement trajectory data. The data is then compared and optimized in a real-world scenario, and eye movement trajectories that do not conform to the visual field settings are removed. This includes the following steps:

[0062] After multiple data acquisition steps in the second step, raw eye movement trajectory information is obtained. This raw trajectory information is then input into two different generative adversarial networks (GANs): a gaze GAN simulates the subtle movements of gaze behavior, and a saccade GAN simulates the rapid movements of saccade behavior. Both GANs include a generator and a discriminator. The generator creates synthetic gaze trajectories, while the discriminator distinguishes between real and synthetic data. First, the generator creates synthetic data based on the raw gaze point data. This synthetic data is used to train the discriminator. The loss from the generated data points is used to adjust the generator's weights. The generator and discriminator compete against each other, optimizing their respective parameters through gradient descent. This makes the samples generated by the generator increasingly realistic, while the discriminator becomes increasingly accurate.

[0063] The generator projects noise vectors to a higher dimension through fully connected layers, then synthesizes gaze trajectory sequences through batch normalization and activation functions. These sequences are then rearranged to accommodate the dwell times required for different eye-movement behaviors. Following the reshaping layer are three deconvolutional blocks to recover the original output, avoiding undesirable boundary effects or information loss, thus obtaining more accurate sequence data.

[0064] The discriminator uses the gaze trajectory sequence as input and can distinguish whether the sequence is a real sequence or a sequence generated by the generator. The discriminator consists of three convolutional blocks, normalization, and activation functions. The input of the last convolutional block is fully expanded and fed into the fully connected layer. The activation function distinguishes between real and generated sequences.

[0065] In the process of synthesizing gaze sequences, each sequence starts with the first gaze position obtained by the Generative Adversarial Network (GAN) based on simulated gaze behavior. Each subsequent gaze position is added to the current sequence. The position between two gaze clusters determines the amplitude of the saccade behavior. At the same time, the saccade behavior has a certain directionality. When adding a sequence, the current sequence needs to be rotated appropriately to achieve correct trajectory synthesis. By repeating the above process, the gaze trajectory can be synthesized.

[0066] For synthesized eye-tracking trajectories, comparative learning can be used to optimize and remove eye-tracking trajectories that do not conform to regional settings in scenarios such as reading and driving. The comparative learning technique employs a self-supervised learning method without labeled data, aiming to distinguish between similar and dissimilar sequences. The comparative learning process can optimize the synthesized eye-tracking trajectories, resulting in synthesized eye-tracking trajectories with high accuracy and robustness.

[0067] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content can be implemented in other ways in the several embodiments provided in this application. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing eye-tracking trajectories for individuals with visual field defects, used for eye-tracking trajectory synthesis and data augmentation in individuals with visual field defects, characterized in that, Includes the following steps: Step 1: Setting the field of view; The visual field area of ​​people with visual field defects is obtained by visual field measurement and visual field expansion rules, and the line of sight plane is calibrated by multiple generated reference calibration points within the visible area. Step 2: Data Generation; Acquire gaze data within the visual field, and use continuous principal component clustering to distinguish gaze data into fixation data and saccade data. Combine time information and landing point region judgment to generate raw eye movement trajectory data; The data generation includes the following steps: Step 21: After acquiring gaze data, classify and cluster eye movements; analyze the principal components of the clusters to obtain sub-clusters, until the lowest-level gaze point is obtained, thus obtaining all gaze behaviors; Step 22: Perform corresponding processing according to the area where the landing point is located; the area where the landing point is located includes: the visible field of view area, the invisible area, the extended field of view area, and the overlapping field of view area; For the visible visual field, eye movement behavior is divided into fixation behavior and saccade behavior; No processing is performed on areas that are not visible to the naked eye; For expanding the visual field, saccades are considered fixation behavior; For overlapping visual fields, fixation behavior between the visible visual field and the extended visual field is considered as saccade behavior. Step 23: Obtain eye movement behavior; the eye movement behavior includes: fixation behavior between visible areas, saccade behavior within the visible area, fixation behavior between the same extended areas, saccade behavior between different extended areas, and saccade behavior between the visible area and the extended area; Step 24: After classifying the eye-tracking behavior, the original motion trajectory is synthesized using the time information recorded by the eye-tracking method; Step 3: Adversarial learning; Train a gaze generation adversarial model and a saccade generation adversarial model based on the original eye movement trajectory information. Generate gaze data and saccade data through the two models respectively. Connect the generated data clusters to form eye movement trajectory data. Compare and learn the data in the actual scene to optimize it, and remove eye movement trajectories that do not conform to the visual field settings.

2. The eye-tracking trajectory synthesis method for people with visual field defects according to claim 1, characterized in that, The setting of the field of view includes the following steps: Step 11: Use a perimeter to measure the user's horizontal and vertical field of vision angles, as well as the expected distortion or clipping area, to obtain field of vision measurement data; Step 12: Convert the field of view measurement data into grayscale values ​​or grayscale levels in the field of view coordinate system; Step 13: Measure the distance from the user to the screen, and use trigonometric functions to calculate the calibration mapping of the human eye's field of vision on the screen to obtain the approximate areas of the visible and invisible regions; Step 14: Smooth the mapped region of the visible area using Gaussian filtering; Step 15: Use morphological erosion to narrow the boundaries of the field of view; Step 16: After obtaining the visible area, invisible area, and extended area, you can mark them on the screen; Step 17: After marking the visual field areas, perform line of sight calibration; set the reference calibration point in the middle of the visible visual field area. If there are multiple visible visual field areas, select the largest visible visual field area; if the visible visual field area is less than 5% of the total visual field area, use the center point of each visible visual field area as the reference calibration point. Step 18: After selecting the reference calibration point, the user is asked to look at the reference calibration point in sequence; at the same time, the coordinates of the reference calibration point and the human eye characteristics are recorded during the looking process, and the eye-tracking method is used to complete the eye-tracking mapping between the human eye and the screen. Step 19: Repeat step 18 until all angles are calibrated, and during the repetition, rotate the rectangle 45° clockwise each time.

3. The eye-tracking trajectory synthesis method for people with visual field defects according to claim 2, characterized in that, The selection criteria for the reference calibration point are as follows: take the rectangle with the largest area in the visible field of view, and the four vertices and the midpoint of the rectangle are the reference calibration points.

4. The eye-tracking trajectory synthesis method for people with visual field defects according to claim 2, characterized in that, The field of view measurement data is represented in the form of angles or positional information relative to the center of the field of view.

5. The eye-tracking trajectory synthesis method for people with visual field defects according to claim 1, characterized in that, The eye movement behaviors include fixation behaviors and saccade behaviors; the fixation behaviors are manifested as dense clusters of points, and the saccade behaviors are the movement from one cluster of points to another. The movement between different fixation behaviors is the saccade behavior.

6. The eye-tracking trajectory synthesis method for people with visual field defects according to claim 1, characterized in that, The adversarial learning includes the following steps: Step 31: Input the original trajectory information into the gaze generation adversarial model and the saccade generation adversarial model respectively; both the gaze generation adversarial model and the saccade generation adversarial model contain a generator and a discriminator; Step 32: The generator creates synthetic data based on the original gaze point data, which is used to train the discriminator; Step 33: Synthesize gaze trajectory sequences through batch normalization and activation functions; each sequence starts from the first gaze position obtained by the Generative Adversarial Network (GAN) from simulated gaze behavior, and each subsequent gaze position is added to the current sequence. The position between two gaze clusters determines the amplitude of the saccade behavior. Step 34: The discriminator uses the gaze trajectory sequence as input to distinguish whether the gaze trajectory sequence is a real sequence or a sequence generated by the generator; Step 35: Perform comparative learning optimization and delete eye-tracking trajectories that do not conform to the regional settings.

Images