Eye movement analysis using co-clustering hidden Markov model (co-clustering EMHMM) and switching hidden Markov model (EMSHMM)

Abstract

Provided are an eye movement analysis using an co-clustering-based hidden Markov model (EMHMM) and an eye movement analysis using a switching hidden Markov model (EMSHMM) to analyze eye movement data in cognitive tasks involving stimuli with different feature layouts and changing cognitive states. A switching hidden Markov model (SHMM) is provided to capture the cognitive state transitions of participants during the task, and an EMSHMM is provided to evaluate preferences in decision-making tasks involving two or more cognitive states. This EMSHMM provides a quantitative measure of individual differences in cognitive behavior / styles, having a significant impact on the interdisciplinary study of cognitive behavior using eye tracking.

Description

Background Art

[0001] Recent studies have shown that people have specific eye movement patterns that are consistent across different stimuli and tasks in visual tasks (e.g., Andrews & Coppola, 1999; Castelhano & Henderson, 2008; Poynter, Barber, Inman, & Wiggins, 2013; Kanan, Bseiso, Ray, Hsiao, & Cottrell, 2015). These specific eye movement patterns may reflect individual differences in cognitive styles or abilities. For example, studies have found that in tasks of viewing scenes, participants who showed stronger curiosity fixated significantly more than those who showed weaker curiosity. In addition, when viewing faces, people with more extroverted and likable personalities fixated on the eyes of others significantly more times than those with less extroverted and likable personalities (Wu et al., 2014).

[0002] In recent years, attempts have been made to use machine learning methods to infer the characteristics of observers from eye movement data (e.g., Kanan et al., 2015). These studies typically use classifiers to discover important eye movement features for distinguishing between two or more observers, but are unable to obtain the overall eye movement pattern associated with a specific observer. To better understand the association between eye movement patterns in visual tasks and individual differences in cognitive styles or abilities, the inventors recently developed a novel eye movement data analysis method, namely eye movement analysis using the hidden Markov model (EMHMM), which takes into account individual differences in the temporal and spatial dimensions of eye movements (Chuk, Chan, & Hsiao, 2014. The EMHMM Matlab toolbox can be obtained from the following website: http: / / visal.cs.cityu.edu.hk / research / emhmm / ).

[0003] The EMHMM method is based on the assumption that during a visual task, the current region of interest (ROI) being fixated depends on the previously fixated ROI. Thus, eye movements in a visual task can be regarded as a Markov random process, and a hidden Markov model (HMM), which is a time series statistical model in machine learning, can be used to better understand this process. Specifically, in this method, the HMM is used to directly model the eye movement data, and the hidden states of the HMM correspond to the ROIs of the eye movements. The transition probabilities between multiple hidden states (ROIs) represent the temporal patterns of eye movements between multiple ROIs. To account for individual differences, one HMM is used to model the eye movement patterns of a person during a visual task based on the person-specific ROIs and two of the transitions between multiple ROIs. The variational Bayesian method, which can automatically determine the optimal number of ROIs, is used to estimate an individual's HMM from the individual's eye movement data. In addition, individuals' HMMs can be clustered according to their similarity (Coviello, Chan, & Lanckriet, 2014) to reveal common patterns. A likelihood measure, which is calculated as the log-likelihood of the data of one model being generated by another model, can be used to quantitatively evaluate the differences between different models.

[0004] Therefore, the EMHMM method is particularly suitable for examining individual differences in eye movement patterns and their associations with other cognitive measures. In addition, since the HMM is a probabilistic time series model, it can function well with a limited amount of data (e.g., 20 trials), but is not suitable for deep learning methods that require a large amount of data for effective training. Thus, the EMHMM method is particularly suitable for psychological research where data availability is limited or data collection is time-consuming.

[0005] The EMHMM method has been successfully applied to face recognition research and has yielded new findings that could not be revealed by other methods to date. For example, two common eye movement patterns for face recognition have been discovered: one is the "holistic" pattern, in which participants mainly fixate on the center of the face, and the other is the "analytical" pattern, which involves more frequent eye movements between the eyes and the center of the face (e.g., see Figure 1a). Interestingly, the analytical pattern tends to have better recognition performance and is applicable to different cultural groups and age groups (e.g., Chuk, Chan, & Hsiao, 2017; Chuk, Crookes, Hayward, Chan, & Hsiao, 2017; Chan, Chan, Lee, & Hsiao, 2018). In addition, participants who used the same eye movement pattern when viewing their own and other-race faces (75%) were significantly more than those who used different eye movement patterns (Chuk, Crookes et al., 2017). In contrast, only about 60% of the participants used the same eye movement pattern during face memorization and face recognition, and their recognition performance was not significantly different from that of those who used different eye movement patterns. This goes against the saccade path theory, which holds that eye movements during face memorization must be reproduced during face recognition for successful recognition (Chuk et al., 2017).

[0006] In addition, the study also found that the elderly used the holistic pattern, while the young used the analytical pattern (e.g., see Figure 1b). This difference was not easily observable from the group eye fixation heatmaps, which demonstrated the ability of the EMHMM method ( Figure 1c ; Chan et al., 2018). Among the elderly, the holistic pattern was associated with a lower cognitive status as assessed by the Montreal (Montreal) Cognitive Assessment (HK-MoCA; Yeung, Wong, Chan, Leung, & Yung, 2014); particularly in terms of executive function and visual attention ability (assessed by the Tower of London (TOL) and Trail Making Test). Interestingly, when the model of the common pattern discovered was used to evaluate the eye movement patterns of new participants when viewing new facial images, this association was replicated, suggesting the possibility of developing representative models in the population for the purpose of screening for cognitive impairment.

[0007] Previous methods for analyzing eye movement data, including using predefined regions of interest (ROIs) or fixation heatmaps, did not consider the temporal information of eye movements, and the use of predefined ROIs may introduce experimenter bias. Additionally, in these methods, analysis is typically performed on average or grouped data, thus failing to reflect individual differences in eye movement patterns. Moreover, there is no quantitative measurement of eye movement pattern similarity that takes into account both spatial (eye fixation positions) and temporal information (fixation order).

[0008] Furthermore, previous EMHMM methods were limited to visual tasks with the same feature layout of stimuli (such as face recognition) and did not address changes in cognitive states. Therefore, new methods that consider both spatial and temporal information are needed to quantitatively measure eye movement patterns, and through such methods, it is possible to more comprehensively predict human behavior based on eye movement patterns. Summary of the Invention

[0009] Methods and systems are provided for analyzing and modeling cognitive tasks involving changes in cognitive states to predict subject behavior and / or cognitive processes. Specifically, eye movement analysis using co-clustering hidden Markov models (EMHMMs) and eye movement analysis using switching hidden Markov models (EMSHMMs) are provided. The models of the present invention summarize the eye movements of an individual by using hidden Markov models, thereby measuring differences in eye movement patterns, being able to discover common eye movement patterns, and providing a quantitative measurement of eye movement pattern similarity that takes into account both the temporal information and spatial information of eye movements. The advantages of the present invention are that the EMSHMM of the present invention enables data analysis and modeling of cognitive tasks involving changes in cognitive states and provides better prediction and analysis of subject behavior and cognitive processes. Additionally, the co-clustering of EMHMM enables data analysis in tasks and cognitive behaviors involving stimuli with different layouts, including but not limited to website browsing, use of information systems, and visual search. Brief Description of the Drawings

[0010] Figure 1a- Figure 1c Detection of two common eye movement patterns in face recognition is shown. Figure 1a shows the overall eye movement pattern on the left and the analyzed eye movement pattern on the right. Figure 1b shows the correlation between overall eye movement and analyzed eye movement with age. Figure 1c Shows the fixation heatmap of older participants on the left and the fixation heatmap of younger participants on the right;

[0011] Figure 2 An example of the co-clustering EMHMM of the present invention is shown. The circles represent ROIs;

[0012] Figure 3Shows a histogram of the symmetric KL (SKL) divergence between the first and second sets of strategies on vehicle or animal images;

[0013] Figure 4A , Figure 4b1- Figure 4b4 , Figure 4C Shows an example of an eye movement strategy. Figure 4A Shows an example stimulus. Figure 4b1- Figure 4b shows the eye movement strategies of the two groups in the face of the stimulus. According to the measurement of the symmetric KL divergence (SKL), the two groups use significantly different strategies on the first two images and less different strategies on the last two images. Figure 4C Shows the fixation heatmaps of the eyes of the two groups on the first image. The different region of Group 1 is orange and that of Group 2 is blue;

[0014] Figure 5a- Figure 5f Shows a correlation analysis. Figure 5a shows the relationship between the EF level and the average fixation count. Figure 5f Shows the correlation between the EF level and the average saccade length. Figure 5c shows the correlation between the EF level (scale) and the preference rating. Figure 5d shows the correlation between the EF level and the scene recognition performance (represented by d'). Figure 5e shows the correlation between the EF level and the lateral inhibition consistency Acc (Flanker Congruent Acc). Figure 5f Shows the correlation between the EF level and the TOL planning time before execution;

[0015] Figure 6 Shows an exemplary SHMM model that summarizes the eye movement patterns of participants in the preference decision task. The blue arrows represent the transition probabilities between cognitive states (higher-level states); the red arrows represent the transition probabilities between ROIs (lower-level states);

[0016] Figure 7 Shows the gaze cascade plots of two participant groups (Group A and Group B) and all participants (All) in the last 2.5 seconds before the response. The red, blue, and green stars at the top indicate the time points when the gaze cascade effect occurs; the black stars at the bottom indicate the time points when there are significant differences in the proportion of time spent by the two groups on the selected items;

[0017] Figure 8A-8B Shows the difference in the transition between the exploration period and the preference-biased period between the two participant groups. Figure 8A Shows the average probability that the two groups are in the preference-biased period throughout the experiment. Figure 8BShows the probability distribution of the number of fixations during the exploration period (upper figure) and the probability distribution of the number of fixations during the preference bias period (lower figure);

[0018] Figure 9A-Figure 9B Shows the average inference accuracy of two participant groups. Figure 9A Shows the average inference accuracy using partial fixations in the trials, which is the percentage of the duration of each experiment from the start. Figure 9B Shows the average inference accuracy of two participant groups using different window lengths from the start of the trial;

[0019] Figure 10A-Figure 10B Shows the average inference accuracy using different fixation patterns. Figure 10A Shows the average inference accuracy of two groups of participants using fixations in the last 2 seconds before making the participant response. Figure 10B Shows the relationship between the eye movement pattern of the participants (measured on an AB scale) and the corresponding inference accuracy; and

[0020] Figure 11 Shows the average inference accuracy using SHMM and conventional HMM at the last 2 seconds of using the trial. Detailed Description

[0021] Provides methods and systems for analyzing and modeling cognitive tasks involving changes in cognitive states to predict subject behavior and / or cognitive processes.

[0022] In some embodiments, provides eye movement analysis of a co-clustering hidden Markov model (EMHMM), where EMHMM is combined with co-clustering of data mining techniques to detect groups of participants with consistent eye movement patterns in individual stimuli for tasks involving different feature layouts. In a particular embodiment, the model is applied to scene perception.

[0023] In some embodiments, provides eye movement analysis using a switching hidden Markov model (EMSHMM) to detect groups of participants with consistent eye movement patterns as the cognitive state changes. The EMSHMM of the present invention is particularly useful for tasks involving decision-making, in which participants need to explore different options and decide which option they prefer; a process involving at least the following two cognitive states: exploration and decision-making.

[0024] Also provides a method for determining the perceptual style and cognitive ability of a subject.

[0025] The EMHMM method of the present invention takes into account the individual differences in eye movements in the temporal and spatial dimensions and uses HMM to model eye movement data, where the hidden states of the model directly correspond to the regions of interest (ROI) of eye movements.

[0026] In some embodiments, the method uses HMMs to summarize each person's eye movement patterns in terms of both personal specific ROIs and the transitions between ROIs, and clusters the individual HMMs based on their similarity. The differences between the models are quantitatively evaluated using a likelihood metric, which reflects the similarity between individual patterns.

[0027] In some embodiments, the EMHMM of the present invention is applied to facial recognition to evaluate the cognitive state in executive and visual attention functions and to evaluate cognitive decline or deficits. For example, using the EMHMM method of the present invention, a perceptual style can be assigned to an individual based on the eye movements of the individual measured and evaluated using the EMHMM of the present invention. A particular eye movement pattern measured using the EMHMM of the present invention is associated with an analytic perceptual style, while other eye movement patterns are associated with a holistic perceptual style. In addition, the analytic perceptual style is associated with improved recognition performance, while the holistic perceptual style is associated with cognitive decline in executive function and visual attention. Thus, the method of the present invention can be used to detect an individual's cognitive performance or decline in cognitive performance based on the individual's eye movement pattern. Importantly, through the method of the present invention, such an assessment of cognitive performance can be made without the need to determine additional physiological parameters. Thus, the method of the present invention is applicable to applications where eye movement detection is the only available data, such as driver assessment and / or eye movement assessment via any computer or electronic device screen.

[0028] In some embodiments, the co-clustering EMHMM of the present invention is used to evaluate individuals with similar eye movement patterns under various stimuli. For example, the EMHMM of the co-clustering method of the present invention includes generating individual HMMs that include the transition probabilities between personalized ROIs and ROIs; determining common patterns by clustering; quantifying the similarity between patterns using the data log-likelihood; and using the similarity metric to examine the relationship between the eye movement pattern and the cognitive metric.

[0029] In some embodiments, the EMHMM under the co-clustering method of the present invention evaluates the eye movement patterns of an individual for each stimulus during scene perception and defines the association between the eye movement pattern and foreground object recognition performance and cognitive ability. In the EMHMM under the co-clustering method of the present invention, the co-clustering formula ensures that the group of participants remains consistent in the face of all stimuli, and the quantitative evaluation using the log-likelihood metric enables the determination of the relationship between the eye movement pattern and various cognitive metrics. Using the method of the present invention, the type of scene image that causes large individual differences in the eye movement pattern can be determined, and the object recognition performance and cognitive ability of an individual can be measured based on the eye movement pattern during scene viewing.

[0030] In some embodiments, the EMHMM of the present invention uses an HMM to model eye movements, where the hidden states of the HMM are directly associated with the regions of interest (ROIs) of the eye movements, and the temporal dynamics of the eye movements are determined based on the transitions between the ROIs and the ROIs.

[0031] In some embodiments, the HMMs of individuals are clustered to detect representative strategies in the subjects, and co-clustering is used to cluster the subjects into groups according to whether the subjects use similar eye movement strategies when facing various stimuli. Subsequently, the symmetric KL (SKL) divergence between the group HMMs is used to quantify the differences between the group strategies. The correlation between the eye movement patterns and the recognition performance and / or cognitive ability is determined by using the clustering score to quantify the similarity between the patterns of the subjects and the group strategies, where the log-likelihood of the subject data under different group strategies is adopted. For example, the co-clustering using data mining techniques clusters the eye gazes of individuals on each image into two groups, thereby generating two sets of HMMs for each image. A histogram of the SKL divergences between the two general pattern groups is generated, where the SKL quantifies the differences between the two group patterns for each stimulus. Preferably, images that cause large differences between individuals are used because larger individual differences are more likely to provide a sufficient amount of differences between individuals for identifying atypical patterns.

[0032] In a preferred embodiment, cognitive ability tests are used together with eye movement pattern detection. For example, the Tower of London (TOL) test is used to measure executive function, the Flanker Task test is used to measure visual attention, and the Verbal&Visuospatial Two-Back Task test is used to measure working memory. Using these methods of the present invention, it has been determined that the eye movement patterns in scene perception are particularly suitable for measuring visual attention and executive function. Therefore, in a preferred embodiment, the co-clustering EMHMM method of the present invention is used to measure the visual attention and executive function of an individual, where a focused eye movement pattern indicates better visual attention and executive function compared to an exploratory eye movement pattern. In a further preferred embodiment, the co-clustering EMHMM method of the present invention is used to measure object recognition performance, where an exploratory eye movement pattern indicates better object recognition performance compared to a focused eye movement pattern.

[0033] In some embodiments, the hierarchical HMM has at least two layers: a high-level HMM that acts as a switch for capturing the transitions between cognitive states; and several low-level HMMs for learning the eye movement patterns of the cognitive states.

[0034] Advantageously, the model of the present invention measures the differences in eye movement patterns by summarizing the eye movements of an individual using a Hidden Markov Model, enabling the discovery of common eye movement patterns and providing a quantitative measure of the similarity of eye movement patterns that takes into account the temporal and spatial information of eye movements. For example, the EMSHMM of the present invention enables data analysis and modeling of cognitive tasks involving changes in cognitive states and provides better prediction and analysis of the behavior and cognitive processes of subjects. In addition, the method of the present invention enables data analysis in tasks and cognitive behaviors involving stimuli with different layouts, including but not limited to website browsing, information system use, visual search, driving, and other complex tasks.

[0035] In a particular embodiment of the present invention, data can be received from any type of eye tracking mechanism (whether optical, electrical, magnetic, or other mechanism for calculating the fixation position of a human eye). In a preferred embodiment, an Eyelink 1000 eye tracker is used to record eye movements, but any eye tracking mechanism can be used to implement the present invention without departing from the spirit or scope of the present invention. In a preferred embodiment, video camera images are used to collect eye tracking patterns.

[0036] In some embodiments, the data received from the eye tracking mechanism is used in a Hidden Markov Model. A Hidden Markov Model is a statistical model mainly used to recover data sequences that cannot be immediately observed. The model derives the probability values of the unobservable data sequence by interpreting other data that depends on the unobservable data sequence and can be immediately observed.

[0037] In some embodiments, the Hidden Markov Model of the present invention represents the visual output (e.g., the raw data received from the eye tracking mechanism) as a randomized function of the invisible internal state (e.g., the cognitive state of the subject).

[0038] The Hidden Markov Model can be used to model the changes in visual attention or eye movements corresponding to the transitions of cognitive states during complex cognitive tasks.

[0039] In a particular embodiment of the present invention, the HMM is directly used to model eye movements, and the hidden states of the HMM are directly associated with the ROIs of eye movements. This enables the determination of the temporal dynamic changes of eye movements based on the transitions between ROIs and the individual-specific ROIs.

[0040] Advantageously, the new method of the present invention detects multiple cognitive states that occur during a task and the eye movement patterns associated with each cognitive state; thus enabling a better understanding of the individual differences in eye movement patterns of complex cognitive tasks in real life and predicting cognitive states from eye movement patterns.

[0041] In some embodiments, the present invention provides a Switching Hidden Markov Model (SHMM). The SHMM belongs to hierarchical HMMs and has two layers, including a high-level HMM and a number of low-level HMMs. Each low-level HMM can be used to learn the eye movement patterns of cognitive states. The high-level HMM acts as a "switching machine" that captures the transitions between cognitive states. It captures the transitions of cognitive states by learning the transitions between low-level HMMs.

[0042] The SHMM of the present invention is used to model the transitions between cognitive states in a cognitive task and the associated eye movement patterns.

[0043] In some embodiments, the high-level states represent cognitive states, while the low-level states correspond to ROIs of stimuli.

[0044] In some embodiments, facial preference decisions are used to model eye movements in a preference decision task, where in each trial, two facial images are presented to a participant and the participant is asked to judge which one they prefer more. According to an embodiment of the present invention, at least two different eye movement patterns are observed. One has no fixation preference for either stimulus, which is related to exploration and information sampling, while the other shows a higher percentage of fixations on the stimulus to be selected, which both reflects and shapes the participant's preference.

[0045] In certain embodiments, the SHMM is used to capture the dynamic changes in cognitive states and eye movement patterns. In some embodiments, one SHMM for each individual is trained to summarize the eye movement behavior of the individual during the task. In some embodiments, data selected from two different participants are used to train two or more SHMMs.

[0046] In a preferred embodiment, individuals are clustered according to the similarity of their SHMMs to discover common eye movement patterns in the task. Different eye movement patterns are associated with different decision-making behaviors and can be detected and measured using the methods and systems of the present invention. Thus, the methods of the present invention can be used to infer, for example, decision-making behaviors from eye movement patterns.

[0047] In some embodiments, the EMHMM using co-clustering is used to discover common patterns in a group of participants and analyze tasks or cognitive behaviors involving stimuli with different layouts, which include but are not limited to website browsing, information system use, and visual search. Other tasks or cognitive behaviors that can be analyzed using the methods of the present invention are reading, picture viewing, video viewing, scene viewing, driving, navigation, etc.

[0048] Advantageously, the methods of the present invention enable the evaluation of more than one cognitive state in a complex task and enable the identification of the cognitive states based on the measured eye movement patterns.

[0049] In a preferred embodiment, a specific eye movement pattern of a specific cognitive state is used to identify the specific cognitive state in a subject.

[0050] In a further embodiment, a system is provided for performing the method of the present invention. In some embodiments, the system includes a camera and a processor configured to detect the eye positions in a facial image captured by the camera. In some embodiments, the processor is further configured to perform the steps and / or algorithms of the present invention. In a preferred embodiment, the processor is configured to read and process data from the camera according to a Matlab toolbox to analyze HMM, EMHMM, and EMSHMM.

[0051] In some embodiments, an electronic device is provided, which includes a camera and a processor configured to detect a central position of an eye in a facial image in response to detecting the eye in the facial image captured via the camera; determine an eye gaze position based on the central position and the pupil position; analyze the eye gaze positions in continuously captured facial images; and measure an eye movement pattern based on the eye gaze positions of the continuously captured facial images.

[0052] In a specific embodiment, the processor is further configured to generate an exploration transition matrix and Gaussian emissions for each subject. In some embodiments, the processor is further configured to cluster or summarize HMMs into separate groups using the Variational Hierarchical Expectation Maximization algorithm, where the HMMs are clustered according to the probability distribution of the HMMs. In other embodiments, the processor is further configured to absorb data input by an external user to associate certain eye movement patterns with certain external task criteria or cognitive state criteria.

[0053] In a further embodiment, the processor is configured to determine the number of fixations of eye movements. Advantageously, according to the eye movement pattern and the fixation probability distribution, the system of the present invention can calculate the probability that an individual is in a certain cognitive state.

[0054] In some specific embodiments, the system of the present invention can be used to infer an individual's preference for a certain choice based on the eye movement pattern.

[0055] In some embodiments, the system of the present invention can be used to determine an individual's cognitive style based on the individual's eye movement pattern. In a specific embodiment, the processor of the system is configured to calculate the log-likelihood of the eye movement pattern, and if a certain cognitive style matches the eye movement pattern of a representative group of individuals having the cognitive style, assign the certain cognitive style to the measured eye movement pattern.

[0056] Further provided is a method of using an eye movement pattern to infer an individual's preference choice. In a particular embodiment, the individual's preference choice is inferred based on the individual's eye movement fixations measured during different time periods from the start to the end of the trial. In a particular embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 25% of the time period from the start of the trial. In a preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 15% of the time period from the start of the trial. In a more preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 10% of the time period from the start of the trial. In a most preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 5% of the time period from the start of the trial.

[0057] Further provided is a method of using an eye movement pattern measured during different time periods before a selection response to infer an individual's preference choice. In a particular embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 10 seconds before the selection response. In a preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 8 seconds before the selection response. In other preferred embodiments, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 6 seconds before the selection response. In a more preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 4 seconds before the selection response. In a most preferred embodiment, the individual's preference choice is inferred by the individual's eye movement fixations measured within the last 2 seconds before the selection response.

[0058] Advantageously, the method enables the measurement of individual differences in eye movement patterns and cognitive styles during complex cognitive tasks. In addition, the method enables the measurement of an individual's preference earlier than previously used methods. Specifically, the method of the present invention enables the measurement of an individual's choice preference using only 75% of the fixations before making a choice.

[0059] Advantageously, using the method of the present invention, particularly using EMSHMM, the accuracy of inferring an individual's preference choice is significantly higher than using, for example, EMHMM.

[0060] In addition, the co-clustering EMHMM of the present invention can be used to infer an individual's cognitive style and / or cognitive ability.

[0061] Importantly, the method of the present invention uses EMSHMM and / or co-clustering EMHMM to infer an individual's preference choices and cognitive style / ability solely from eye gaze information, without the need for additional physiological information, making the method and system of the present invention particularly suitable for situations where the only information available from an individual is eye movement measurements, to determine the individual's cognitive style, infer the individual's preference choices, and / or measure the individual's cognitive ability.

[0062] All patents, patent applications, provisional applications, and publications mentioned or cited herein, including all drawings and tables, are hereby incorporated by reference in their entirety to the extent that they are not inconsistent with the express teachings of this specification.

[0063] The following are examples that illustrate the procedures for practicing the present invention. These examples should not be construed as limiting. Unless otherwise specified, all percentages are by weight and all solvent mixture ratios are by volume.

[0064] Materials and Methods

[0065] Example 1 - Eye Movement Analysis Using an Hidden Markov Model (EMHMM) in Facial Recognition

[0066] Using the EMHMM method, the inventors previously detected global and analytic eye movement patterns in facial recognition by clustering the eye movement patterns of 34 young and 34 elderly individuals (Chan, Chan, Lee, and Hsiao (2018)). In the representative model, the ellipses represent ROIs corresponding to 2-D Gaussian emissions (Figure 1a). The prior value represents the probability that a trial starts from an ellipse / ROI. The transition probability represents the probability of observing a specific transition from the current ROI to the next ROI. The small images on the right show the assignment of actual fixations to the ROIs and the corresponding fixation heatmaps. Note that the clustering algorithm (Coviello, Chan, & Lanckriet, 2014) summarizes each model in a cluster into a representative HMM using a pre-specified number of ROIs, which may result in ROI overlap. Here, the number of ROIs in the representative model is set to 3, which is the median value of the number of ROIs in each model. Figure 1b shows the frequencies of young and elderly individuals adopting global and analytic patterns: significantly more elderly individuals adopt the global pattern, and more young individuals adopt the analytic pattern. Figure 1c Group fixation heatmaps for the elderly and young are shown.

[0067] Example 2 - Use of EMHMM under Different Stimuli

[0068] Since the EMHMM method that provides a quantitative measure of individual differences in eye movement patterns is limited to tasks where the stimuli have the same feature layout (e.g., faces), a new method has been developed that combines EMHMM with co-clustering, a data mining technique, to detect groups of participants who have consistent eye movement patterns when faced with stimuli that involve different feature layouts. This new method is applied to scene perception. Using the new EMHMM method with co-clustering of the present invention, exploratory (switching between foreground and background information) and focusing (primarily on the foreground) eye movement strategies can be detected. The exploratory pattern is associated with better foreground object recognition performance, while people using the focusing pattern have better feature integration ability in the lateral inhibition task and more efficient planning ability in the Tower of London (TOL) task. Advantageously, these new methods can be used to employ eye tracking as a window into understanding individual differences in cognitive abilities and for screening cognitive deficits.

[0069] Scene perception involves complex and dynamic perceptual and cognitive processes that are influenced by the observer's goals and different scene attributes at multiple levels. Eye movements during scene perception reflect the complexity of the cognitive processes involved and can thus potentially provide rich information about individual differences in perceptual styles and cognitive abilities. Therefore, it is not yet clear whether eye movement patterns can always reflect cultural differences in perceptual styles and individual differences in cognitive abilities.

[0070] Previous studies have attempted to use machine learning methods to infer observer characteristics from eye movement data. These studies typically use classifiers to discover important eye movement features that can distinguish between two or more observers. However, classifiers can only find important features that distinguish observers and do not provide any information about the eye movement patterns associated with a particular observer. The inventors have developed the EMHMM method, where HMM is a time series statistical model. This method takes into account the individual differences in eye movements in both the temporal and spatial dimensions. EMHMM assumes that during a visual task, the current region of interest (ROI) being fixated depends on the previously fixated ROI. Thus, eye movements can be considered a Markov random process, and this process can be better understood using HMM. Other studies have used HMM / probability models to model eye movements / visual attention and cognitive behavior, where the hidden states of the model represent cognitive states. In contrast, the EMHMM of the present invention directly uses HMM to model eye movement data, and the hidden states of the model directly correspond to the ROIs of eye movements. To account for individual differences, HMM is used to summarize the eye movement patterns of each individual based on the individual-specific ROIs and the transitions between ROIs. A variational Bayesian method that can automatically determine the number of ROIs is used to estimate the eye movement patterns of each individual from the individual data. The HMMs are clustered based on their similarity to reveal common strategies. The differences between the models are quantitatively evaluated using a likelihood metric, which can reflect the similarity between individual patterns. Therefore, this method is particularly suitable for examining individual differences in eye movement patterns and their associations with other cognitive measures. In addition, since EMHMM belongs to the Bayesian probability model, it can also operate well with only a limited amount of data; in contrast to deep learning methods that require a large amount of data to be effectively trained. The EMHMM of the present invention has been applied to face recognition research and has obtained new findings that have not been revealed by existing methods to date. For example, two common eye movement strategies have been identified: analytical (more focused on the eyes) and holistic (focused on the nose) (Figure 1a). The moderating effect of culture is small (Chuk et al., 2017). The analytical pattern is associated with better recognition performance, indicating that the retrieval of diagnostic information (i.e., the eyes) can better predict performance (Chuk, Chan, & Hsiao, 2017). In contrast, compared to younger people, older people adopt the holistic pattern, and their holistic pattern is associated with a lower cognitive state, especially in terms of executive and visual attention functions: the more the pattern is biased towards the holistic (focused on the nose), the lower the cognitive state (Chan, Chan, Lee, Hsiao, 2018). This correlation reappears when new participants view new facial images using the group HMMs found from older participants, indicating the possibility of developing representative HMMs for cognitive screening through a large population.Similarly, the impaired ability to judge facial expressions in insomnia patients is associated with the use of less eye-focused patterns, suggesting that impaired visual attention control may be the reason for the weak emotional facial perception ability in insomnia patients (Zhang et al., 2019). These results consistently suggest the possibility of using eye tracking to screen for the assessment of cognitive decline or deficits.

[0071] EMHMM has been restricted to tasks involving stimuli with the same feature layout, such that the discovered ROIs correspond to the same features for each individual stimulus. For tasks with stimuli having different layouts, it has been proposed to include the perceived image as a feature in the ROI representation. However, although this method has improved ROI discovery, it does not work well when the features between stimuli are significantly different, such as in scene perception.

[0072] Example 3 - Novel EMHMM Using Co-Clustering

[0073] The present invention provides a new method that uses an HMM to model the eye movements of each participant while viewing a specific stimulus and uses data mining techniques, co-clustering (e.g., see Govaert & Nadif, 2013), to detect subjects who share similar eye movement patterns across the stimuli. The co-clustering formula ensures that the groups of subjects remain consistent for all stimuli. The result is a grouping of the subjects and their representative HMMs for each stimulus ( Figure 2 ). As in the existing method, a log-likelihood metric is used to quantitatively evaluate the similarity between an individual's eye movement pattern and the representative HMM across the stimuli. These results are used to examine the relationship between the eye movement patterns and other cognitive metrics. Using this method, individual differences in eye movement patterns during scene viewing were examined, and it was determined (1) what types of scene images (animals in nature vs. vehicles in the city) caused greater individual differences in eye movement patterns, and (2) how the eye movement patterns during scene viewing were associated with the subsequent foreground object recognition performance and cognitive abilities of the subjects.

[0074] In one study, 61 Asian participants aged 18 - 25 years (M = 20.77, SD = 1.70) were recruited from the University of Hong Kong (35 of whom were female). All participants had normal vision or corrected-to-normal vision. The viewing materials included 150 scene images of animals in natural environments and 150 scene images of vehicles in urban environments. Images with different numbers of foreground objects, feature layouts, and foreground object positions were used to increase stimulus variability to provide sufficient opportunities to elicit individual differences in eye movement patterns.

[0075] The scene perception task included a passive viewing phase and a surprise recognition phase (see Chua et al., 2005 for details). In the passive viewing phase, for each scene type (animals and vehicles), participants viewed 60 pictures for 5 seconds each and rated their liking for each picture on a scale of 1 to 5. In the surprise recognition phase, for each scene type, participants viewed 60 images of foreground objects that they had previously viewed, with half presented in the same background as before and the other half presented in a new background, along with the same number of lure images with new backgrounds and new objects. Participants saw one image at a time and judged whether they had seen these foreground objects during the passive viewing phase. The image remained on the screen until the participant responded. In both phases, animal and vehicle scene images were presented in two separate blocks, and the block order was balanced across participants. An EyeLink 1000 eye tracker was used to record participants' eye movements, and a chin rest was used to minimize head movement as much as possible. Each trial started with a fixation cross at the center of the screen. When a stable fixation was observed at the fixation cross, the experimenter started presenting the image. Then, an image was presented at the center of the screen, and the viewing distance for the participant was 0.6 m, with a viewing angle of 35°×27°. A 9-point calibration procedure was performed before presenting each block. Whenever the offset correction error exceeded 1° viewing angle, recalibration was performed.

[0076] In addition, participants performed 3 cognitive tasks to examine whether their eye movement patterns were related to their cognitive abilities:

[0077] (1) The Tower of London (TOL) task for testing executive function / planning ability (see, e.g., Phillips et al., 2001): In each trial, participants saw 3 beads randomly placed on 3 pegs as the starting position and the target position. They were required to move one bead at a time and place the beads at the target position as quickly as possible with the fewest number of moves, and plan in their minds before making the move. There were a total of 10 trials. The total number of extra moves, the number of correct attempts, the total planning time before making the first move, and the total execution time of the moves were measured.

[0078] (2) The flanker task for testing selective attention (see, e.g., Ridderinkhof, Band, & Logan, 1999): Participants judged the direction of an arrow whose flankers were 4 other arrows. In the incongruent condition, the flanker arrows pointed in the same direction as the target arrow, while in the congruent condition, the flanker arrows pointed in the opposite direction. In the neutral condition, the flankers were non-directional symbols.

[0079] (3) Verbal and visuospatial dual-back task for evaluating working memory capacity (see, e.g., Lau et al., 2010): In each trial, the participant judges whether the presented symbol / symbol location is the same as the symbol / symbol location presented 2 trials before in the verbal / visuospatial task.

[0080] The eye movements of the participants during the passive viewing phase were analyzed using co-clustering EMHMM (adapted from http: / / visal.cs.cityu.edu.hk / research / emhmm). HMM was used to summarize the eye movements of each participant while viewing each stimulus. The HMMs of individuals while viewing each stimulus were clustered to discover 2 representative strategies among the participants. Then, using co-clustering, the participants were clustered into 2 groups according to whether they used similar eye movement strategies when facing each stimulus ( Figure 2 ). The difference between the group strategies corresponding to each stimulus was quantified by the symmetric KL (SKL) divergence between the two groups of HMMs to examine whether natural or artificial images could cause greater individual differences in eye movement patterns. SKL is given by (KL 12 + KL 21 ) / 2, where KL 12 is the KL divergence between the HMMs of group 1 and group 2 obtained using the data of group 1 (Chuk, Chan, & Hsiao, 2014), and KL 21 is the opposite (KL is asymmetric). By comparing the SKL metrics between natural and artificial images, the characteristics of the images that usually result in a greater SKL were determined.

[0081] It was further examined whether the performance of the two groups of participants in foreground object recognition and cognitive tasks was different. To examine the correlation between eye movement patterns and recognition performance / cognitive ability, the similarity between the eye movement patterns of the participants and the group strategies was examined using the clustering score CS = (|L1| - |L2|) / (|L1| + |L2|), where L1 and L2 are the log-likelihoods of the participant data generated under the group 1 and group 2 strategies, respectively (Chan et al., 2018). A larger positive value of CS indicates a higher similarity to group 1, and a smaller negative value of CS indicates a higher similarity to group 2.

[0082] Example 4 - Determining eye movement strategies in scene perception using co-clustering EMHMM

[0083] The EMHMM using co-clustering of 120 image stimuli was used to cluster the data obtained from 61 participants as in Example 3 into two groups. Group 1 included 37 participants and Group 2 included 24 participants. The co-clustering model estimated two HMMs for each image stimulus, corresponding to the strategies of Group 1 and Group 2. For each image, the differences between different strategies were measured in SKL. As Figure 3 shown, compared with vehicle images, animal images caused greater differences in eye movement strategies between the two groups, t(118) = -5.626; p < 0.0001. Figure 4A Four example images of the two groups and their corresponding HMMs are shown. Figures 4b1 and 4b2 show two examples of eye movement strategies with large SKL differences between the two groups. In particular, Group 2 paid more attention to foreground objects, while Group 1 explored the image, looking at both foreground objects and the background. In addition, when observing animals, Group 2 paid more attention to the eyes, while Group 1 focused on the nose ( Figure 4C ). Figures 4b3 and Figure 4b4 show examples where the two groups had similar eye movement strategies. Generally speaking, for images where foreground objects (animals or cars) are more prominent than the background, such as animals in the woods or cars on the road, the differences in eye movements were larger. This indicates that animals are more prominent than vehicles, and animal images usually can cause greater differences in eye movement strategies than vehicle images. In contrast, images with cluttered backgrounds and foreground objects that participants are not interested in usually trigger similar eye movement strategies. The strategies of Group 1 and Group 2 are called exploratory strategy and focused strategy respectively, and the clustering score between the two strategies is called the exploratory-focused (EF) score. In fact, compared with participants using the focused strategy (Table 1 below), participants using the exploratory strategy had more average fixation times, t(59) = 3.793, p <.001, and longer saccade lengths, t(59) = 4.881, p <.001, and their eye movement patterns were more similar to the exploratory strategy (EF score), with more average fixation times, r(60) =.477, p <.001, and longer saccade lengths, r(60) =.544, p < 0.001 ( Figure 5a- Figure 5e). Since the EMHMM does not use sequence length information, naturally, the difference in average fixation times appears as a result of clustering. These results are consistent with the explanation that Group 1 is more exploratory than Group 2.

[0084]

[0085] Table 1: Comparison of eye movement statistics and cognitive task performance between the exploratory strategy and the focused strategy (FOR: Foreground Object Recognition).

[0086] Example 5 - Association between EMHMM using co-clustering and eye movement patterns and recognition performance / cognitive abilities

[0087] Participants using an exploratory strategy also had significantly lower image preference scores, t(59) = -2.689, p =.009; better foreground object recognition in d’, t(59) = 3.434, p =.001; lower accuracy in the consistency trials of the flanker task, t(59) = -2.121, p =.038; and longer total planning time before execution in the TOL task, t(58) = 2.395, p =.02 (Table 1); (TOL results for one participant were missing due to technical problems). Note that participants had a significantly greater advantage in foreground object recognition performance when the foreground object was presented in the original background than when it was presented in a new background, F(1,59) = 9.126, p =.004, although this advantage was significant in both cases (Table 1). Consistent with these findings, correlation analysis indicated that the similarity between participants’ eye movement patterns and the exploratory strategy (EF rank) was negatively correlated with the preference rank, r(60) =.301, p =.018; positively correlated with scene recognition performance in d’, r(60) =.381, p =.002; and slightly positively correlated with the TOL planning time before execution of the action, r(59) =.223, p =.08( Figure 5a- Figure 5e). Taken together, these findings suggest that the exploratory strategy is associated with lower image preference ratings, better scene recognition performance, reduced facilitation from consistent surrounding cues in the flanker task, and less efficient planning in the TOL task.

[0088] Overall, for images containing animal faces, participants using a focused strategy (analytical style) looked more at the eyes of the animal faces, indicating the mobilization of local attention. In contrast, participants using an exploratory strategy (holistic style) tended to look more at the center of the animal faces, indicating the mobilization of global processing. This analytical style is related to focusing on foreground objects in scene perception and on the eyes in face recognition. However, when considering individual differences, culture has little modulation on eye movement patterns (Chuk et al. 2017).

[0089] It has been observed that, regardless of whether the foreground object appears in a new or old background, participants using an exploratory strategy have better foreground object recognition performance than those using a focusing strategy; this advantage is greater when the foreground object appears in an old background than in a new background. This finding suggests that, during scene perception, a more exploratory eye movement strategy may be beneficial for remembering foreground objects, as more extraction cues can be obtained through exploration. Consistent with this suggestion, it has been shown that associative processing is inherent in scene perception, indicating that an exploratory strategy may facilitate associative processing and thus enhance scene memory. This finding contrasts with the relevant literature on face recognition, in which an eye focusing strategy (related to the involvement of local attention and the focusing strategy found in this study) yields better recognition performance due to better retrieval of diagnostic features (the eyes).

[0090] Conversely, using the method of the present invention, it has been shown that participants using a focusing strategy perform better in the congruency trials of the lateral inhibition task and spend less pre-planning time in the TOL task than participants using an exploratory strategy. The advantage observed in the congruent (or non-incongruent) or neutral trials of the lateral inhibition task suggests that these participants may have better feature integration ability. In the TOL task, the shorter pre-planning time, with no differences in the number of moves, number of correct trials, or execution time, indicates that the participants have more efficient planning ability. Thus, because participants using a focusing strategy prefer to look at the eyes of animals, the focusing strategy may be related to the more eye-focused strategy in face recognition, which is associated with better face recognition performance, visual attention (trajectory making task), and executive function (TOL task; Chan et al., 2018).

[0091] Using the method of the present invention, it has also been observed that images with foreground objects prominent relative to the background are prone to cause large individual differences in eye movement patterns, and animal images cause greater individual differences than vehicle images ( Figure 3 ). This phenomenon may be due to the category-specific attention of humans to animals, making animals more prominent than vehicles or other object types, providing better opportunities to elicit differences between exploratory and focusing strategies. This finding has important implications for the possibility of using eye tracking as a screening tool for cognitive impairment, as images that cause greater individual differences will be more likely to provide sufficient differences between individuals to identify atypical patterns.

[0092] In summary, the novel EMHMM using co-clustering method of the present invention can effectively summarize and quantitatively evaluate individual differences in eye movement strategies in tasks involving stimuli with different feature layouts, and thus obtain new findings not yet discovered by existing methods. By applying this new method to scene perception, exploratory and focused strategies of Asians were detected. The exploratory strategy was associated with better foreground object recognition performance, while the focused strategy was associated with better feature integration and planning ability. In addition, images with foreground objects prominent relative to the background caused greater individual differences in eye movement patterns. These results have important clinical and educational implications for using eye tracking in cognitive deficit detection and cognitive performance monitoring. Advantageously, the new EMHMM method using co-clustering method of the present invention can be applied to a variety of visual tasks and helps to understand cognitive behavior using eye movement patterns.

[0093] Example 6 - Eye movement analysis using switched hidden Markov model (EMSHMM) in the detection of different cognitive states using face preference decision-making tasks

[0094] In previous probabilistic methods for modeling visual attention in complex cognitive tasks, the hidden states of the model represented cognitive states, thus capturing the temporal dynamics of cognitive state transitions, but not the dynamics of eye movements. To measure multiple cognitive states that occur during the task and the eye movement patterns associated with each cognitive state, an EMHMM with hierarchical HMM was used. For example, a hierarchical HMM with two layers contains a high-level HMM and several low-level HMMs. Each low-level HMM acts as a "switching machine" to capture transitions between cognitive states. This is achieved by learning the transitions between low-level HMMs. This method of the present invention is called switched HMM (SHMM). The SHMM of the present invention is used to capture the dynamic changes of cognitive states and eye movement patterns, following the existing EMHMM method, and a SHMM for each participant is trained to summarize the eye movement behavior of the participant during the task. Then, the individual SHMMs are clustered according to the similarity of eye movement behavior to detect common eye movement patterns in the task and to examine different eye movement patterns associated with different decision-making behaviors.

[0095] Applying the novel EMSHMM of the present invention, eye movement data was collected from a facial preference decision-making task, and two participant groups were compared. The preference decision-making task was a two-alternative forced-choice task and consisted of two parts. In the first part, the participants were shown 120 computer-generated bald faces (female and male). The participants were asked to rate the attractiveness of these faces from 1 to 7. These ratings were used for stimulus pairing in the second part, and no eye movement analysis was performed. After the first part was completed, faces of the same gender with similar ratings were paired into 60 pairs as the stimuli for the second part. In the second part, in each trial, each pair of faces was shown on the screen, one on the left and one on the right. The participants were asked to indicate which face they preferred. There was no time limit. The participants could freely move their eyes to compare the two images. The participants were told to press a button once they made a decision to indicate which face they preferred (left or right). Eye movements were recorded using an Eyelink 1000 eye tracker. During data collection, fixation position information was extracted using the Eyelink Data Viewer. The saccade movement threshold was 0.15 degrees of visual angle; the saccade acceleration threshold was 8000 degrees per square second; the saccade speed threshold was 30 degrees per second. These are the EyeLink default values for cognitive research.

[0096] Example 7 - Switching Hidden Markov Model

[0097] The standard Hidden Markov Model (HMM) includes a vector of prior values of hidden states, a transition matrix of hidden states, and Gaussian emissions for each hidden state. The prior values represent the probability that the time series data starts from the corresponding hidden state. The transition matrix represents the transition probability between any two hidden states. The Gaussian emissions represent the probability association between the observed time series data and the hidden states. In the previous EMHMM method (Chuk et al., 2014), the hidden states corresponded to regions of interest (ROIs), the emissions corresponded to eye fixation positions, and the emissions in the ROIs were represented as 2-D Gaussian distributions (see Figure 1a).

[0098] Compared with the standard HMM, the Switching HMM (SHMM) includes a two-level hidden state sequence; the low-level hidden state sequence models the temporal pattern of the time series data that follows the standard HMM, while the high-level hidden state sequence indicates the transitions between the HMM parameters used by the low-level hidden state sequence. Formally, z n,t ∈{1,…,K} is the low-level hidden state, s n,t ∈{1,…,S} is the high-level hidden state, x n,t is the observed value, where n represents the sequence and t represents the time. Both the high-level state sequence and the low-level state sequence are first-order Markov chains. The prior probability and transition of the low-level hidden state depend on the current high-level state,

[0099]

[0100] where τ n is the length of the n-th sequence. The high-level state sequence is parameterized by the prior vector ρ and the transition matrix B,

[0101] p(s n,1 = j) = ρ j , p(s n,t = j′|s n,t-1 = j) = b j,j′ (3)

[0102] Assume the high-level state is s n,t = j, the low-level state sequence is parameterized by the prior vector π (j) and the transition matrix A (j) to parameterize the low-level state sequence,

[0103]

[0104] The emission density is Gaussian and depends only on the low-level hidden state (i.e., it is shared among high-level states),

[0105]

[0106] where μ k , is the mean vector and covariance matrix of the Gaussian distribution. Assume the number of low-level states for each high-level state is the same. The joint probability model of the SHMM is

[0107]

[0108] In practice, by combining the high-level and low-level hidden state variables into a single hidden state variable whose value is the Cartesian product of the low-level and high-level state values, the SHMM can be converted into a standard HMM. Here, assume the low-level states are shared among high-level states (S), so the number of low-level states (K) is the same for each high-level state. Thus, the equivalent standard HMM has S*K extended hidden states The extended state takes on the value pair (j,k), where j represents the high-level state and k represents the low-level state. The transition probability and prior value are thus defined as,

[0109]

[0110] Therefore, the relationship between the extended hidden state and the two separate levels of the hidden state sequence is defined as

[0111]

[0112] The transition matrix and prior vector have a block structure,

[0113]

[0114] In this implementation, the high-level hidden state represents the cognitive state, while the low-level hidden state corresponds to the ROI for the stimulus. The high-level state sequence has its own transition matrix that controls the transitions between cognitive states. The low-level states (ROIs) are shared between the high-level states. Taking the preference decision-making task as an example, assume that the participant has two cognitive states: the exploration period and the preference bias period. The exploration period involves information sampling without preference for a specific stimulus, and the preference bias period forms a preference, and the eye movement behavior will be affected by the preference. Assume a simplified decision-making process in which the participant starts from the exploration period. Once the participant has sampled enough information in the exploration period, they transition to the preference bias period and cannot transition back to the exploration period. That is, once entering the preference bias period, one cannot exit until a decision-making response is made. To examine the gaze cascade effect, i.e., the gaze bias for the later-selected object (e.g., see Shimojo et al. 200), assume that the low-level HMM has two ROIs, each corresponding to the selected stimulus. Figure 6 An example model summarizing the participant's eye movement pattern is shown. As shown, the high-level HMM contains two cognitive states as its hidden states: the exploration period and the preference bias period. The blue arrows represent the transitions between the two states, and the numbers represent the transition probabilities. The eye movements in each state are modeled by a low-level HMM, and the hidden states of the HMM represent the ROIs of the eye movements. The red arrows represent the transitions between the ROIs. The ROIs of these two cognitive states are the same, but the transition probabilities are different.

[0115] Example 8 - Training an individual SHMM

[0116] The Expectation-Maximization (EM) algorithm is executed to estimate the SHMM parameters. In the Expectation step (E-step), using the standard forward-backward algorithm, the responsibilities are calculated through the block transition matrix, the initial state vector, and the emission density. In the Maximization step (M-step), the prior and pairwise responsibilities are summed over the high-level and low-level states respectively to produce the parameter updates for the high-level and low-level states.

[0117] For example, the prior responsibilities are summed over the low-level hidden states of each high-level state to produce the parameter updates for the low-level state sequence, and then the prior responsibilities are summed over the high-level states to produce the parameter updates for the high-level state sequence. Similarly, the pairwise responsibilities are summed over the low-level hidden states of each high-level state to produce the updates for each low-level transition, and then the pairwise responsibilities are summed over the high-level hidden states to produce the update for the switching (transition) matrix of the high-level state sequence.

[0118] Two SHMMs are trained for each participant; one using data from the trials selected from the left side and the other using data from the trials selected from the right side. As shown below, for each individual, the two SHMMs are combined into one. Preliminary analysis shows that the exploration periods of the trials selected from the left side and the right side are similar. In other words, regardless of which side is selected, the eye movements during the exploration period are consistent. Therefore, the transition matrices of the exploration periods of the models selected from the left side and the right side are directly put together and averaged. For the preference bias period, the average values of the two preference side parameters and the two non-preference side parameters are calculated, essentially normalizing the trials selected from the right side to the trials selected from the left side. To focus the analysis on the transitions between stimuli during the preference decision, for the low-level states, each model uses only two Gaussian emissions, one for each side ( Figure 6 ). For SHMM estimation, one Gaussian centered on each stimulus is initialized, with a covariance that covers the stimulus. Thus, the low-level states of the model can be considered pre-specified (and thus not hidden) here because it is possible to determine with high confidence which stimulus is being viewed. The advantage of using Gaussian emissions instead of discrete emissions is that Gaussian emissions can be easily extended to an analysis that explores more ROIs (i.e., more hidden low-level states) for each stimulus. Two high-level hidden states are used to reflect the transition of the participant from the exploration period to the preference bias period during the trial.

[0119] For SHMM estimation, the transition matrix of the high-level state is initialized to [0.95, 0.05; 0.0, 1.0], and the high-level prior is initialized to [1.0, 0.0], which encodes the hypothesized behavior of starting from the exploration period and staying in the exploration period (0.95), and then finally transitioning to the preference bias period (0.05) without returning (0.00). During training, this initialization keeps the probability of transitioning from the preference bias period to the exploration period at 0 (because all potential sequences of transitioning from the preference bias period to the exploration period are given a probability of 0). The transition matrix of the low-level state is initialized to a uniform distribution. After initialization, the Gaussian ROIs and transition probabilities are updated in the EM algorithm.

[0120] Example 9 - Clustering to Discover Common Patterns

[0121] To study the general eye movement patterns adopted by all individuals during the exploration period, the exploration transition matrices and Gaussian emissions of each individual are used to create HMMs, and the variational hierarchical expectation maximization (VHEM) algorithm is used to cluster or summarize these HMMs into a group (e.g., see Coviello et al., 2014). The VHEM algorithm clusters the HMMs into a predefined number of groups according to the probability distribution of the HMMs and uses representative HMMs to characterize each group. A similar procedure is carried out to obtain the general eye movement patterns of the preference bias period.

[0122] To study the individual differences in decision-making behavior, the k-means clustering algorithm was used to cluster the participants' high-level transition matrices into two groups (see, e.g., MacQueen, 1967). The aim was to discover the differences in high-level cognitive behavior between the exploration period and the preference bias period. For each group, the representative exploration-period HMM and preference-bias-period HMM were calculated by running the VHEM algorithm on the exploration-period HMM and preference-bias-period HMM of that group, respectively. 1 Then, the differences in decision-making behavior between the two groups of participants were studied, including their differences in the gaze cascade graph / effect, the transitions between the exploration period and the preference bias period, the distribution of the number of fixations within the exploration / preference bias period, and the accuracy of inferring preference choices from eye movement data.

[0123] Example 10 - Transitions between the exploration period and the preference bias period

[0124] After the participants' high-level transition matrices were clustered into two groups, the differences in decision-making behavior between the two groups of participants were further studied. More specifically, the probability that a participant was in the preference bias period from the start to the end of the trial was studied. For each trial, the posterior probabilities of all possible high-level state sequences were calculated based on the given observed eye fixation data. Then, the probability sequence of being in the preference bias period was calculated by computing the expectation of the high-level state sequence, i.e., by computing the weighted average of all high-level state sequences, where the weights were the posterior probabilities. The above procedure was carried out for all trials of all participants. Since the duration of each trial was different, to study the proportion of time that participants spent in the exploration period and the preference bias period relative to the entire trial duration, the different durations of each trial were standardized by dividing each trial into the same number of segments (21 segments in this experiment). Then, for each segment in the trial, the probability of the participant being in the preference bias period was calculated. For each participant, the average probability of each segment in all trials was calculated. Finally, the averages of the same group of participants were averaged, and the average probability of each trial segment was plotted for the two groups of participants separately. This graph represents the percentage of time that participants were in the preference bias period in each trial segment. This study enabled the detection of inter-group temporal dynamic differences in cognitive state changes during the trial.

[0125] The number of fixations in the exploration period and the preference bias period between the two groups was also studied. To this end, for each trial and each participant, the posterior probabilities of all high-level state sequences were calculated, and then the number of fixations in the exploration period and the preference bias period was calculated. Then, the aggregated probabilities of all trials and all high-level sequences were used to form the probability distribution of the number of fixations of the participant in each high-level state. Then, the probability distributions of each group of participants were averaged.

[0126] Example 11 - Inference of Individual Preferences

[0127] It was investigated whether SHMM can be used to infer individual preference choices in an experiment. For each participant, the trials in the preference decision task were divided into two groups: one group was the trials where the left image was chosen as the preference, and the other group was the trials where the right image was chosen. Similar to the preference decision task, the facial images used in the experiment were matched in terms of gender and attractiveness rating. Thus, both the left and right sides could be equally chosen. For each group, all but one trial were used to train the left - chosen and right - chosen SHMMs, while the hold - out trial was used for testing. For testing, an aggregated SHMM was created from the two trained SHMMs, which could be used to infer the participant's choice. Specifically, the aggregated SHMM contained 3 high - level states: exploration period, left - choice preference - bias period, and right - choice preference - bias period. In the high - level transition matrix of this aggregated SHMM, the transition probabilities from the exploration period to the preference - bias periods were equally distributed between the left - choice preference - bias period and the right - choice preference - bias period. Finally, to infer the participant's choice by testing the eye - fixation sequence, the posterior probability of the high - level state of the last fixation of a given test sequence was calculated, which indicates the probability that the participant was in the left - choice preference - bias period or the right - choice preference - bias period at the end of the trial. The left / right - choice preference - bias period with the higher probability was predicted as the participant's choice. The above procedure was repeated for each participant and for all trials to calculate the inference accuracy.

[0128] The inference was conducted in three ways. First, to study the percentage of fixations required to make inferences above the chance level in the experiment, the first 10% of the fixations were used for inference, and this proportion was increased by 5% successively until all the fixations (100%) were used. Thus, the inference task was conducted 19 times, and the average inference accuracy was calculated each time. Second, to study how long it takes to infer a decision, the increasing duration of the fixation sequence was inferred from the start of the trial (e.g., fixations in the first 1 second, fixations in the first 2 seconds, etc.). Third, the gaze - cascade model suggests that although during the trial, preferences are formed when the participant switches fixations between two stimuli, the fixations just before the end of the trial are usually significantly biased towards the stimulus preferred by the participant. Thus, these fixations should be able to predict the participant's preference better than the previous fixations. Therefore, in a separate study, only the fixations in the last 2 seconds before the decision was made were used to infer the preference.

[0129] Example 12 - Classification of Individual SHMMs

[0130] Train an SHMM for each participant and create: 1) a standard HMM that uses the exploration period transition matrix to represent the eye movement pattern of the participant during the exploration period, and 2) a standard HMM that uses the preference bias period transition matrix to represent the eye movement pattern of the participant during the preference bias period.

[0131] Table 2 below shows the average high-level state transition matrix. The participant starts from the exploration period and has a 55% probability of remaining in the exploration period. There is also a 45% probability of transitioning to the preference bias period and remaining there until the end of the trial.

[0132] Table 2. High-level state transition matrix for all subjects.

[0133] Explore Preference Prior value 1.00 0.00 Explore 0.55 0.45 Preference 0.00 1.00

[0134] Next, use the VHEM algorithm to cluster the exploration period HMMs of 24 participants into a representative HMM. Tables 3a and 3b below show the transition matrices of the representative exploration period HMM and preference bias period HMM, respectively.

[0135] Table 3a. Transition matrix of the representative exploration period HMM obtained by clustering 24 exploration period HMMs into a group.

[0136] Left Right Prior value 0.70 0.30 Left 0.64 0.36 Right 0.12 0.88

[0137] Table 3b. Transition matrix of the representative preference bias period HMM obtained by clustering 24 preference bias period HMMs into a group.

[0138]

[0139]

[0140] As shown in Table 3a above, the participant tends to first observe the left side with several fixations and then transition to observing the right side with several fixations. After viewing the right side, the participant rarely goes back to fixate on the left side (12% probability). This indicates that the participant quickly scanned both sides during the exploration period.

[0141] Similarly, Table 3b above shows that participants in the preference bias period are more likely to preferentially maintain fixation on the side to be chosen (77%), while the probability of fixating on the side not to be chosen is smaller (67%); in addition, participants in the preference bias period are more likely to transition from the side not to be chosen to the side to be chosen (33%), while the probability of transitioning from the side to be chosen to the side not to be chosen is smaller (23%).

[0142] To study the individual differences in the gaze cascade effect, the participants were divided into two groups according to their high-level transition matrices. It was found that one group (Group A) consisted of 11 participants and the other group (Group B) consisted of 13 participants. Tables 4, 5a, and 5b below show the high-level transition matrices of the two groups, as well as the transition matrices of the representative exploration period and the preference bias period HMMs.

[0143] Table 4. Transition matrices of the high-level states for Group A (11 participants) and Group B (13 participants).

[0144] Group A Explore Preference Prior value 1.00 0.00 Explore 0.68 0.32 Preference 0.00 1.00

[0145] Group B Explore Preference Prior value 1.00 0.00 Explore 0.45 0.55 Preference 0.00 1.00

[0146] Table 5a. Transition matrices of the representative exploration period for Group A (11 participants) and Group B (13 participants).

[0147] Group A Left Right Prior value 0.76 0.24 Left 0.67 0.33 Right 0.17 0.83

[0148] Group B Left Right Prior value 0.64 0.36 Left 0.60 0.40 Right 0.09 0.91

[0149] Table 5b. Transition matrices of the representative preference bias period for Group A (11 participants) and Group B (13 participants).

[0150] Group A Selected Not selected Prior value 0.50 0.50 Selected 0.83 0.17 Not selected 0.25 0.75

[0151] Group B Selected Not selected Prior value 0.54 0.46 Selected 0.71 0.29 Not selected 0.39 0.61

[0152] From the high-level state transition matrix, the probability that Group A remained in the exploration period (68%) was higher than that of Group B (45%). Therefore, the exploration period of Group A was longer than that of Group B. By studying the transition matrix during the exploration period, it was found that Group A (76%) was more likely to start exploring from the left side than Group B (64%), and the probability that Group A maintained fixation on the left side (67%) was greater than that of Group B (60%). After switching to the right side, the probability that Group A switched back to the left side was also greater than that of Group B (17% vs. 9%).

[0153] During the preference bias period, the participants in Group A showed a significant fixation bias to stay on the chosen side. More specifically, the participants in Group A (83%) were more likely to keep fixating on the side they were going to choose than Group B (71%), and the probability of switching between the two sides was smaller than that of Group B.

[0154] Example 13 - Cascade Diagram

[0155] Analysis showed that during the preference bias period, participants tended to gaze more at the side they were going to choose. However, the clustering results showed that for the participant group (Group A), this difference was more obvious compared to the other group (Group B). To visualize the difference in the gaze cascade effect between the two participant groups, a gaze cascade plot was generated. This plot shows the probability that participants gazed at the image they were going to choose within the last 2.5 seconds before the response. Figure 7 The gaze cascade plot showing the two groups and their averages.

[0156] As Figure 7 shown by "All" in , towards the end of the trial, participants spent more time looking at the side they were going to choose. The proportion of time looking at the chosen side steadily increased from a probability level of 0.5 until it reached around 0.87. The probability that each participant gazed at the chosen stimulus at each time point was estimated at 100 - millisecond (ms) intervals. To test the hypothesis that there were significant differences in the probability of gazing at the chosen side at different time intervals and that there were differences between the two groups of participants in terms of the time - interval effect, a mixed ANOVA was conducted on the probability of gazing at the chosen side, with the time interval as the within - subject variable and the group as the between - subject variable. The results showed: a significant main effect of the time interval, F(3.043,66.938) = 52.163, p <.001, η p 2 = 0.703, a significant main effect of the group, F(1,22) = 5.481, p = 0.029, η p 2 = 0.199, and a non - significant interaction between the time interval and the group, F(3.043,66.938) = 2.307, p = 0.084, η p 2 = 0.095 2 . In addition, there was a significant linear trend between the time intervals, F(1,22) = 126.657, p < 0.001, η p 2 = 0.852, and a quadratic trend, F(1,22) = 19.609, p < 0.001, η p 2 = 0.471. These results indicate that during the last 2.5 seconds before the response, the probability that participants gazed at the chosen side increased significantly. In addition, the probability that Group A gazed at the chosen side was higher than that of Group B, indicating a stronger cascade effect.

[0157] Post-hoc t-tests showed that participants were significantly more likely to look at the selected item starting approximately 1100 milliseconds before the end of the trial (i.e., the onset of the gaze cascade effect), t(23) = 2.27, p = 0.033, d = 0.46 (where d refers to Cohen's d, a measure of effect size representing the standardized difference between two means), until the end of the trial. During this time, the probability of looking at the selected face increased from 58% to 87%. Additionally, there was a short period between 2200 and 1600 milliseconds before the end of the trial during which, according to the t-test, the probability that participants looked at the selected side was also slightly higher than chance (about 55%; Figure 7 ).

[0158] Plots of the two participant groups showed some interesting differences. Participants in Group A showed a stronger gaze cascade effect. At the end of the trial, the probability that Group A looked at the selected item reached 94.5%. A t-test was conducted to study when the probability that they viewed the selected stimulus was higher than chance. The results showed that this occurred at approximately 1000 milliseconds before the end of the trial, t(10) = 2.44, p = 0.035, d = 0.74, at which point they spent approximately 66% of their time on the selected item. In contrast, participants in Group B showed a weaker cascade effect, with approximately 81% of their time looking at the selected item at the end of the trial. The t-test indicated that at 900 ms before the end of the trial, the proportion of time viewing the selected item was significantly higher than chance, t(12) = 2.29, p = 0.041, d = 0.64, at which point the probability of fixating on the selected side increased from 59% to 81%. Between 2100 and 1600 milliseconds before the end of the trial, Group B also had a slightly higher than chance fixation on the selected side (54%) for a short period ( Figure 7 ). Independent samples t-tests were also used to compare the proportions of the two groups at each time point. It was found that from 700 ms before the end of the trial to the end of the trial, Group A spent significantly more time looking at what was going to be selected than Group B ( Figure 7 ). Thus, although both groups showed a gaze cascade effect, it differed in both magnitude and onset time, indicating significant individual differences in the gaze cascade effect. These results suggest that the EMSHMM method of the present invention enables the detection of individual differences in the gaze cascade effect by clustering participants' eye movement patterns based on the similarity of their eye movement patterns.

[0159] In addition, using the HMM-based method, the similarity between the eye movement patterns of each participant during the preference bias period and the representative patterns of Group A or Group B can be quantitatively evaluated by calculating the log-likelihood of the eye movement patterns of the participants generated by the representative models. To quantify the eye movement patterns of the participants along the continuum between the representative patterns of Group A and Group B, the A-B rating is defined as (LA–LB) / (|LA|+|LB|), where LA is the log-likelihood of the eye movement patterns generated by the Group A model and LB is the log-likelihood of the eye movement patterns generated by the Group B model (Chan et al., 2018). The larger the A-B rating, the more similar the pattern is to the representative pattern of Group A, and vice versa for the representative pattern of Group B. Among the participants, a significant positive correlation was observed between the A-B rating and the gaze cascade effect (r =.50, p =.012) as measured by the average probability of fixation on the selected item from the onset of the effect (1000 ms before the response) to the end. In other words, during the preference bias period, the more similar the eye movement patterns of the participants are to the representative pattern of Group A, the stronger their gaze cascade effect is.

[0160] Example 14 - Transition between the exploration period and the preference bias period

[0161] To investigate whether the transition behavior between the two cognitive states during the entire trial is different for the two groups of participants, the trial duration was standardized by dividing each trial into 21 time periods, and the percentage of trials for each participant, or the frequency with which the participant was in the preference bias period within each time period, was studied. Figure 8A The average probability of being in the preference bias period during the entire trial for the two groups is shown. Figure 8B The probability distribution of the number of fixations during the exploration period and the probability distribution of the number of fixations during the preference bias period are shown. The vertical lines represent the standard error.

[0162] The results showed that for both groups of participants, the probability of being in the preference bias period increased shortly after the start of the trial. To test the hypothesis that Groups A and B had different probabilities across the time periods, a mixed ANOVA analysis was performed on the probability of being in the preference bias period, with time period as the within-subject variable and group as the between-subject variable. The results showed a significant time period effect, F(1.384, 30.445) = 339.091, p <.001, η p 2 = 0.939, a significant group effect, F(1, 22) = 7.770, p = 0.011, and a significant interaction between group and time period, F(1.384, 30.445) = 10.064, p =.001, η p 2= 0.314. These results indicate that overall, Group B has a higher probability of being in the preference - bias period than Group A, especially during the initial time segments of the experiment ( Figure 8A ). Although Group A enters the preference - bias period later, the gaze - cascade effect of Group A is stronger. In contrast, Group B enters the preference - bias period earlier, but the gaze - cascade effect is weaker.

[0163] A t - test was used to test the hypothesis that the average number of fixations per trial is different between Group A and Group B. It was found that the average number of fixations in the experiment for Group A (M = 29.0) was significantly greater than that for Group B (M = 13.3), t(22)=4.44, p <.001, d = 1.75. The average number of fixations during the exploration period was also greater for Group A (Group A: M = 3.38; Group B: M = 1.88), t(22)=8.46, p <.001, d = 3.37, and the average number of fixations in the preference - bias state was also greater (Group A: M = 25.61; Group B: M = 11.5), t(22)=4.10, p <.001, d = 1.62 ( Figure 8B ). However, after normalizing the total number of fixations, there was no difference in the fraction of fixations during the exploration period or the preference - bias period between Group A (M = 0.147) and Group B (M = 0.173), t(22)= - 1.11; p = 0.28; d = 0.45.

[0164] In addition, during the experiment, the hypothesis that the number of eye - gaze switches between different stimuli is different between Group A and Group B was tested. Group A (M = 5.66) had more switches on average than Group B (M = 4.10), t(22)=2.81, p = 0.01, d = 1.12. However, this effect was mainly due to the fact that generally Group A had more fixations during the experiment. After normalizing the total number of fixations in the experiment, the fraction of eye fixations that switched between stimuli for Group A (M = 0.211) was less than that for Group B (M = 0.328), t(22)= - 7.20, p <.001, d = 2.97. This effect indicates that compared with Group B, Group A explored the previous stimulus for a longer time before switching to another stimulus. It should be noted that since clustering is only based on the cognitive (higher - level) state transitions of the participants, the differences in the number of fixations per trial and the switching frequency between stimuli will naturally result from the clustering.

[0165] Example 15 - Inference of Participant Preference Selection

[0166] It was also explored whether the model could be used to infer the preference selection of individuals in each trial given partial eye - movement data. The first 10% of the fixations of the trials based on the normalized duration were used and gradually increased in 5% steps. Figure 9AShows the average inferred accuracy of the two groups using partial fixations within the percentage of each trial duration from the start of the trial. Figure 9B Shows the average inferred accuracy of two groups using different window lengths from the start of the trial (note that the participants had different trial lengths, with an average length of 5.65 ± 2.57 seconds; the average trial length of Group A was 7.67 ± 2.32 seconds, while the average trial length of Group B was 3.93 ± 1.13 seconds). The red, blue, and green stars at the top represent the data points where the accuracy of each group and all participants was significantly higher than the chance level based on the t-test.

[0167] As Figure 9A shown, when using the first 75% to 90% of the fixations during the trial, the average inferred accuracy of Group B was higher than that of Group A. Conversely, when using the first 95% or all of the fixations of the trial, the inferred accuracy of Group A was higher than that of Group B. To test the hypothesis that Group B had higher inferred accuracy when using 75% to 90% of the fixations, while Group A had higher inferred accuracy when using 95% to 100% of the fixations, a mixed ANOVA analysis was performed on the inferred accuracy, where the amount of fixation (75% to 100%, in 5% steps) was used as the within-subject variable and the group was used as the between-subject variable. The results showed a main effect of the amount of fixation, F(1.356,29.840) = 30.778, p < 0.001, η p 2 = 0.583, and there was an interaction between the amount of fixation and the group, F(1.356,29.840) = 5.657, p = 0.016, η p 2 = 0.205. The main effect of the group was not significant, F(1,22) = 0.077, n.s. These results indicate that, as Figure 9A shown, when using a smaller amount of fixations (75% to 90%), Group B had higher inferred accuracy, while when using a larger amount of fixations, Group A had higher inferred accuracy. When comparing the inferred accuracy with the chance level (0.5) using a t-test, it was found that when using the first 75% to 85% of the fixations, the inferred accuracy of Group B was significantly higher than the chance level, while this was not the case for Group A. When using the first 90% to 100% of the fixations, the inferred accuracy of both groups was significantly higher than the chance level ( Figure 9A ). In other words, the participants in Group B showed a bias towards the preferred stimulus earlier than the participants in Group A.

[0168] To study the actual time when the accuracy of a group was significantly higher than the chance level, Figure 9BThe average inferred accuracy was plotted using the time window (in seconds) from the start of the trial. After the start of the trial, the inferred accuracy of Group B increased more rapidly than that of Group A, was significantly higher than the chance level around 3 seconds, and then reached saturation around 4 seconds. In contrast, the inferred accuracy of Group A increased slowly, reached above the chance level around 6 seconds, and reached saturation around 10 seconds. Although the accuracy of Group A increased more slowly, its value at saturation was higher than that of Group B.

[0169] Example 16 - The last 2 seconds before the response

[0170] The gaze cascade effect suggests that the eye movement pattern just before the preference - determining response may provide strong clues for inferring preferences. According to Figure 7 , participants began to show a tendency to look more frequently at the chosen side approximately 2 seconds before the response. Therefore, the fixation within the last 2 seconds before the response was used to examine the accuracy of inferring participants' preferences. Figure 10A Shows the average inferred accuracy using fixations within the last 2 seconds before the response for the two groups. Figure 10B Illustrates that the more similar the participants' eye movement pattern is to the representative pattern of Group A (i.e., the closer to the right side of the X - axis), the higher the inferred accuracy.

[0171] First, the hypothesis that the average accuracy of inferring the preference decisions of participants in Group A and Group B using the last 2 seconds is significantly higher than the chance level was tested, and the results supported this hypothesis ( Figure 10A ): For Group A, M = 0.93, t(10)=15.89, p <.001, d = 4.79; for Group B, M = 0.71, t(12)=3.79, p =.003, d = 1.05. In addition, another hypothesis that the inferred accuracy of Group A is significantly higher than that of Group B was tested. The results supported this hypothesis, t(22)=3.26, p =.004, d = 1.33. Moreover, during the preference bias period, the more similar the participants' eye movement pattern is to the representative pattern of Group A, that is, the less similar it is to the representative pattern of Group B (measured by the A - B scale), the higher the inferred accuracy (r =.47, p =.02; Figure 10B ). This result is consistent with the observed stronger gaze cascade effect in Group A ( Figure 7 ). The clustering of the two groups is based entirely on eye movement data, so the group differences in inferred accuracy naturally result from the clustering.

[0172] It was also tested whether a conventional HMM that does not infer the cognitive state transitions of participants (such as the HMM used in the previous EMHMM method (Chuk et al., 2014)) could reveal the participants' preference choices. For this purpose, the same inference task was performed using a conventional HMM in the EMHMM method. As Figure 11As shown, for all subjects, the average inference accuracy of SHMM (M = 0.81) was higher than that of HMM (M = 0.64), t(46) = 2.71, p = 0.009, d = 0.78. A two-way ANOVA analysis of the inference accuracy was performed with group and model (SHMM / HMM) as independent variables. The results showed that the main effect of the group was significant, F(1,44) = 5.04, p = 0.03, η p 2 = 0.096, the main effect of the model was significant, F(1,44) = 8.75, p = 0.005, η p 2 = 0.166. There was no interaction between the group and the model, F(1,44) = 1.91, p =.17. This result once again demonstrated the advantage of EMSHMM in modeling eye movement patterns in tasks involving changes in cognitive states. Figure 11 Shows the average inference accuracy of all participants using SHMM and conventional HMM when using the last 2 seconds of the trial.

[0173] Example 17 - Relevance of Research Results

[0174] The present invention proposes a new method, EMSHMM, for modeling eye movement patterns in tasks involving changes in cognitive states. Similar to the previous hidden Markov modeling method for eye movement data analysis, EMHMM, the EMSHMM method has several advantages over traditional eye movement data analysis methods such as ROI or fixation heat map analysis, including the ability to explain individual differences in the spatial and temporal dimensions of eye movements (i.e., by discovering personalized ROIs and transition probabilities between ROIs) and the ability to quantitatively evaluate these differences. Contrary to EMHMM, which uses a single conventional HMM to model eye movements and assumes that the participants' strategies remain consistent throughout the trial, the EMSHMM method uses multiple low-level HMMs corresponding to different strategies / cognitive states, as well as a high-level state sequence, to capture the transitions between different strategies / cognitive states. Therefore, the EMSHMM method is particularly suitable for analyzing eye movement data in complex tasks involving changes in cognitive states, such as decision-making tasks.

[0175] To demonstrate the advantages of using the EMSHMM method, a preference decision task was conducted in which participants viewed two faces with similar attractiveness ratings and decided which one they preferred. Previously, two different eye movement patterns were observed at different stages of the trial; participants typically began by exploring both options and then focused on the preferred one at the end of the trial, and these two eye movement patterns were related to different cognitive states. In the instant EMSHMM method, it was assumed that the two eye movement patterns were respectively related to two cognitive states, namely the exploration period and the preference bias period. A switching HMM (SHMM) was used to summarize the eye movement patterns of participants in the preference decision task. The SHMM included two ROIs corresponding to the two face choices; two low-level HMMs that respectively summarized the eye movement patterns during the exploration period and the preference bias period; and a high-level state sequence that captured the transitions between the two cognitive states.

[0176] The summary of the high-level / cognitive state transitions for all participants showed that, on average, participants had a 55% probability of remaining in the exploration period, a 45% probability of transitioning to the preference bias period, and remaining until the end of the trial (Table 2). When summarizing the exploration period HMMs of all participants in a representative model, it was found that participants had a bias to start by looking at the left-side stimulus and remain on the left for exploration before switching to the right (Table 3a). In contrast, when summarizing the preference bias period HMMs of all participants in a representative model, it was found that participants looked more often at the preferred stimulus to be chosen (Table 3b). When the percentage of time that participants looked at the stimulus to be chosen before the end of the trial was plotted, it showed a steady increase in this percentage at approximately 1.5 seconds before the end, demonstrating the gaze cascade effect ( Figure 7 ).

[0177] When clustering the SHMMs of participants into two groups based on their cognitive state transitions, one group (Group A) showed the gaze cascade effect earlier and with a stronger effect than the other group (Group B; Figure 7 ). Throughout the trial, these two groups of participants also showed interesting differences in the temporal dynamics of their eye movement patterns. More specifically, participants in Group A entered the preference bias period later than Group B ( Figure 8A ), but had a stronger cascade effect. In addition, before using the first 90% of fixations, the preference of Group A for the two options could not be inferred by using the above-chance performance of pre-trial fixations. In contrast, the preference of Group B could be inferred by using the above-chance performance of only the first 75% of fixations ( Figure 9A)。However, when only fixations within the last 2 seconds before the decision response were used, the inference of Group A's preference was more accurate than that of Group B. This phenomenon indicates that although the participants in Group A revealed their preference in the eye movement pattern later than those in Group B during the experiment, their eye movement pattern contains more information for inferring their preference. Recent research has shown that indecision or decision procrastination is related to information tunnel vision: indecisive people tend to collect more information about the item they will ultimately choose while ignoring information about other options. Therefore, in this study, the participants in Group B showed a more "indecisive" eye movement pattern than those in Group A because they entered the preference bias period earlier, stayed in the exploration period for a shorter time, and switched between the two stimulus choices more frequently, which may be characteristics of information tunnel vision. Therefore, there is a relationship between the similarity of the eye movement pattern to the representative pattern of Group B (evaluated using the EMHMM / EMSHMM method) and the personality measurements related to indecision of the participants.

[0178] No literature has pointed out these individual differences in eye movement patterns and cognitive styles during the decision-making process. More specifically, previous studies have only observed that the gaze cascade effect reveals that decision-making is related to the final fixation in the experiment. However, using this method, it can be shown that the inference of the participants' preference is significantly earlier than the display of the gaze cascade effect, and for some participants (e.g., Group B), this inference can achieve above-chance performance using only the first 75% of the fixations. Interestingly, these participants also tend to show a weaker gaze cascade effect. These findings demonstrate the importance of considering individual differences in understanding human decision-making behavior. Importantly, although the conventional HMM without cognitive state transitions in the EMHMM method can also explain the individual differences in eye movement patterns, the accuracy of inferring the participants' preference choices using EMSHMM is significantly higher than that using EMHMM, which is surprising. Advantageously, EMSHMM can better capture the cognitive processes involved in the task, resulting in higher inference accuracy.

[0179] More advantageously, this method can infer the participants' preference only from the eye gaze transition information during the decision-making process. In contrast, previous methods required combining eye movement measurements with other information, such as additional physiological measurements or relevant visual features including integrated skin conductance, blood volume pulse, and pupil response.

[0180] In addition, although previous methods achieved an average accuracy of 81%, this method can infer the participants' preference from only the eye gaze transition information using EMSHMM, and the accuracy exceeds 90%.

[0181] In addition, EMSHMM provides a quantitative measure of the similarity between individual eye movement patterns by calculating the log-likelihood of human eye movement data generated by representative HMMs. For example, studies have shown that the similarity of the eye movement patterns of participants during preference bias to the representative pattern of Group A or their dissimilarity to the representative pattern of Group B (measured on an A - B scale) is positively correlated with the gaze cascade effect and the inferential accuracy using fixations in the last 2 seconds before the response. In addition to studying the relationship between eye movement patterns and other psychometric measures, using the method of the present invention, it is also possible to determine how the eye movement pattern similarity measure is modulated by factors related to decision-making style, such as gender, culture, sleep deprivation, etc. Using EMHMM, it has been shown that during face recognition, the similarity of eye movement patterns to an eye-centered analysis pattern is associated with better recognition performance, while the similarity to a nose-centered holistic pattern is associated with cognitive decline in the elderly (see, for example, Chuk, Chan, et al., 2017; Chuk, Crookes, et al., 2017; Chan et al., 2018), and individuals with insomnia symptoms show eye movement patterns that are more similar to the representative nose-mouth pattern during face expression judgment compared to healthy control groups (Zhang, Chan, Lau, & Hsiao, 2019). Advantageously, the EMSHMM of the present invention can be used to study how eye movement patterns are associated with other psychometric measures and factors that can affect eye movement patterns in more complex tasks involving changes in cognitive states.

[0182] Although the analysis using the method of the present invention has so far focused on the eye gaze transition behavior between two stimulus choices in a preference decision task by using only two ROIs, each ROI corresponding to one stimulus, further methods are provided to analyze the eye movement patterns for each stimulus, such as the transition probability between ROIs (low-level states) and ROIs, to capture individual differences in information extraction in addition to the gaze transitions in decision-making behavior. Previous studies have shown that participants tend to fixate on features or fixation locations during subjective decision-making processes. For example, it has been found that attractive and aversive features receive more attention than those of medium attractiveness, and brands located in the center of the shelves in a store are more likely to be selected by customers. Since individuals differ in how they obtain information from stimulus choices during decision-making or in general cognitive tasks, the method of the present invention uses the EMSHMM toolbox (see Chuk et al., 2014) to capture these individual differences by inferring personalized ROIs for each stimulus using the Gaussian mixture model method, and enables the determination of the optimal number of ROIs for each participant by Bayesian methods. Advantageously, due to the use of a large number of high-level states, the EMSHMM method is able to discover finer-grained cognitive states between the exploration period and the preference bias period in terms of similarity. For example, in more complex cognitive tasks, such as driving or cooking, due to the large number of high-level states, the present method enables the detection of more discrete cognitive states that are important for the task and its associated eye movement patterns.

[0183] SHMM can represent the differences in the transition matrix within a trial (intra-trial differences), while other methods, such as the mixed HMM of Altman (2007), add random effects to the HMM. In particular, random effects are added to the emission density means and the log probabilities of the transition matrix and the priors. This enables the differences between subjects or between trials to be represented by one model. Thus, in some embodiments of the present invention, SHMM is extended to add random effects to model the differences between subjects in a single model.

[0184] In summary, in some embodiments, the novel method of the present invention based on EMSHMM analyzes eye movement data in tasks involving cognitive state changes, where for each participant, SHMM is used to capture data on cognitive state transitions during the task, and HMM is used to summarize the eye movement patterns during each cognitive state.

[0185] In some embodiments, the EMSHMM of the present invention is applied to a facial preference decision-making task. In a specific embodiment, the EMSHMM of the present invention identifies two common eye movement patterns from participants. One pattern enters the preference-biased cognitive state relatively late, shows a strong gaze cascade effect just before the decision response, and enables the determination of the preference decision relatively late in the trial. The other pattern shows the preference decision earlier in the trial, stays in the preference-biased cognitive state for a longer time, and has a weaker gaze cascade effect at the end of the trial, resulting in a lower accuracy of decision response inference.

[0186] These surprising differences are naturally the result of clustering based only on eye movement data, and no existing method in the literature has revealed them. Compared with previous methods, the EMSHMM method of the present invention has unexpected advantages in capturing eye movement behavior in the task and infers the decision responses of participants with higher accuracy. In addition, the EMSHMM provides a quantitative measure of the similarity between individual eye movement patterns, so it is particularly suitable for research on individual differences in cognitive processes using eye movements, and has had a significant impact on interdisciplinary research on cognitive behavior using eye tracking.

[0187] It should be understood that the examples and embodiments described herein are for illustrative purposes only, and those skilled in the art will think of various modifications or changes based on these examples and embodiments, and these modifications or changes will be included within the spirit and scope of this application and the scope of the appended claims. In addition, any element or limitation of any invention or its embodiment disclosed herein can be combined with any and / or all other elements or limitations (individually or in any combination) disclosed herein or any other invention or its embodiment, and all such combinations are considered to be within the scope of the present invention, without limitation thereto.

[0188] References

[0189] Altman, R. (2007). Mixed hidden Markov models: An extension of the hidden Markov model to the longitudinal data setting. Journal of the American Statistical Association, 102, 201 - 210.

[0190] Andrews, T. J., & Coppola, D. M. (1999). Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments. Vision research, 39(17), 2947-2953.

[0191] Castelhano, M. S., & Henderson, J. M. (2008). Stable individual differences across images in human saccadic eye movements. Canadian Journal of Experimental Psychology, 62(1), 1-14.

[0192] Chan, C. Y. H., Chan, A. B., Lee, T. M. C., & Hsiao, J. H. (2018). Eye movement patterns in face recognition are associated with cognitive decline in older adults. Psychonomic Bulletin & Review, 25(6), 2200-2207.

[0193] Chuk, T., Chan, A. B., & Hsiao, J. H. (2014). Understanding eye movements in face recognition using hidden Markov models. Journal of Vision, 14(11):8, 1-14.

[0194] Chuk, T., Chan, A. B., & Hsiao, J. H. (2017). Is having similar eye movement patterns during face learning and recognition beneficial for recognition performance?Evidence from hidden Markov modeling. Vision Research, 141, 204-216.

[0195] Chuk, T., Crookes, K., Hayward, W. G., Chan, A. B., & Hsiao, J. H. (2017). Hidden Markov model analysis reveals the advantage of analytic eye movement patterns in face recognition across cultures. Cognition, 169, 102 - 117.

[0196] Coviello, E., Chan, A. B., & Lanckriet, G. R. (2014). Clustering hidden Markov models with variational HEM. The Journal of Machine Learning Research, 15(1), 697 - 747.

[0197] Govaert, G., & Nadif, M. (2013). Co - clustering: models, algorithms and applications. ISTE, Wiley. ISBN 978 - 1 - 84821 - 473 - 6.

[0198] Kanan, C., Bseiso, D. N., Ray, N. A., Hsiao, J. H., & Cottrell, G. W. (2015). Humans have idiosyncratic and task - specific scanpaths for judging faces. Vision research, 108, 67 - 76.

[0199] Lau, E. Y. Y., Eskes G. A., Morrison, D. L., Rajda, M., Spurr, K. F. (2010). Executive function in patients with obstructive sleep apnea treated with continuous positive airway pressure. J. Int. Neuropsych. Soc., 16, 1077 - 1088.

[0200] MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 281 - 297). Berkeley, CA: University of California Press.

[0201] Phillips, L. H., Wynn, V. E., McPherson, S., & Gilhooly, K. J. (2001). Mental planning and the Tower of London task. Q. J. Exp. Psychol. - A, 54, 579 - 597

[0202] Poynter, W., Barber, M., Inman, J., & Wiggins, C. (2013). Individuals exhibit idiosyncratic eye - movement behavior profiles across tasks. Vision research, 89, 32 - 38.

[0203] Ridderinkhof, K. R., Band, G. P., & Logan, D. (1999). A study of adaptive behavior: effects of age and irrelevant information on the ability to inhibit one's actions. Acta Psychol., 101, 315 - 337.

[0204] Shimojo, S., Simion, C., Shimojo, E., & Scheier, C. (2003). Gaze bias both reflects and influences preference. Nature neuroscience, 6(12), 1317 - 1322.

[0205] Wu, D. W. L., Bischof, W. F., Anderson, N. C., Jakobsen, T., & Kingstone, A. (2014). The influence of personality on social attention. Personality and Individual Differences, 60, 25 - 29.

[0206] Yeung, P. Y., Wong, L. L., Chan, C. C., Leung, J. L., & Yung, C. Y. (2014). A validation study of the Hong Kong version of Montreal Cognitive Assessment (HK - MoCA) in Chinese older adults in Hong Kong. Hong Kong Med. J., 20(6), 504 - 510.

[0207] Zhang, J., Chan, A. B., Lau, E. Y. Y., & Hsiao, J. H. (2019). Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: An eye - tracking study. Sleep, 42(2), zsy220.

Claims

1. A method for determining the cognitive style and / or cognitive ability of a subject, the method comprising: Collect eye movement data of the subjects, wherein the eye movement data has stimuli with different characteristic layouts; Calculate multiple regions of interest (ROIs) based on the eye movement data; Measure the number of transitions between the multiple ROIs; Calculate the transition probability between the multiple ROIs; Cluster the eye movements into groups using co-clustering with an Expectation-Maximization Hidden Markov Model (EMHMM), the Expectation-Maximization Hidden Markov Model (EMHMM) including a vector of prior values of hidden states, a transition matrix of the hidden states; and Gaussian emissions for each hidden state; wherein the prior values indicate the probability that the time series data starts from the corresponding hidden state; the transition matrix indicates the transition probability between any two hidden states; and the Gaussian emissions indicate the probability association between the observed time series data and the hidden states; Use the co-clustered eye movement group model to evaluate the data log-likelihood of each individual; Measure executive function using the Tower of London (TOL), measure visual attention using a lateral inhibition task, and measure working memory using a verbal and visuospatial dual-back task; Determine the relationship between the data log-likelihood measurement and the measured executive function, visual attention, and working memory; and Based on the data log-likelihood measurement, use the co-clustered eye movement group model or the eye movement group to which the eye movements of the subject are co-clustered to determine the cognitive style and / or cognitive ability of the subject.

Images