Ocular system for deception detection

Abstract

To provide a method of deception detection based on ocular information.SOLUTION: The method of deception detection, motion risk assessment, or learning optimization is based on subject's ocular information obtained by providing a video camera configured to record a close-up view of a subject's eye. The ocular information is processed to identify changes in subject's eye signals through use of a convolutional neural network. The changes in the ocular signals are evaluated from the convolutional neural network by a machine learning algorithm. The results can then be presented in terms of a level of a deception, a level of operational risk, or an optimization system to optimize learning. The method is facilitated by identifying at least one predicted distortion identified in a stroma that can be captured using only a visible spectrum camera that is correlated to predicted responses in an iris muscle tissue.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS [Para 1] This international application claims priority to U.S. Non-provisional Utility Patent Application No. 17 / 247,634, filed December 18, 2020, U.S. Non-provisional Utility Patent Application No. 17 / 247,635, filed December 18, 2020, U.S. Non-provisional Utility Patent Application No. 17 / 247,636, filed December 18, 2020, and U.S. Non-provisional Utility Patent Application No. 17 / 247,637, filed December 18, 2020. All of these non-provisional utility patent applications claim priority to U.S. Provisional Application No. 62 / 950,918, filed December 19, 2019, the entire contents of which are hereby incorporated by reference in their entirety.

[0002] [Para 2] The present invention relates generally to ocular systems. More particularly, the present invention relates to ocular systems capable of deception detection, motion risk assessment, and learning optimization enabled by inferring stroma deformation through transillumination of iris muscles. [Background technology]

[0003] [Para 3] The inventors of the present application have extensive experience with the ocular systems disclosed in U.S. Provisional Patent Application No. 62 / 239,840, U.S. Patent No. 10,575,728, issued March 3, 2020, U.S. Patent Application No. 16 / 783,128, filed February 5, 2020, and current U.S. Patent Application Publication No. 2020 / 0170560, the entire contents of which are hereby fully incorporated by reference. Summary of the Invention [Problem to be solved by the invention]

[0004] [Para 4] Thus, there is a need for an improved ocular system. The present invention fulfills these needs and provides other related advantages. [Means for solving the problem]

[0005] [Para 5] Eyeball System for Deception Detection [Para 6] An exemplary embodiment of the present invention is a method for deception detection based on eye information of a subject, the method comprising the steps of: providing a stand-off device configured to view the subject during testing, the stand-off device being in no physical contact with the subject, the stand-off device having at least one video camera configured to record a close-up view of at least one eye of the subject, the stand-off device having or connected to a computing device; providing a cognitive state model configured to determine a high to low cognitive load experienced by the subject, the cognitive load measuring the extent to which the subject utilizes mental resources to formulate his or her response; and providing an emotional state model configured to determine a high to low arousal state experienced by the subject, the arousal state based on activation of the subject's nervous system; and providing a cognitive state model configured to determine a high to low arousal state experienced by the subject, the arousal state based on activation of the subject's nervous system. The method includes the steps of recording eye information of at least one eye of the subject with a camera; establishing a baseline state of eye information of at least one eye of the subject prior to questioning the subject; asking the subject questions and allowing the subject to answer the questions; processing the eye information to identify changes in the subject's eye signals after asking the questions and including time for the subject to answer the questions; evaluating, with a computing device, a cognitive state model and an emotional state model based solely on the changes in the eye signals to estimate a probability that the subject is telling the truth or being deceptive; and determining a binary output of truth or deception. and displaying the binary output to an administrator.

[0006] [Para 7] In other exemplary embodiments, the changes in eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), refixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil sway. The features may include any of the following: eye movement, convergence latency, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stromal change detection, eye region (strabismus), stromal deformation, and iris muscle changes.

[0007] [Para 8] In other exemplary embodiments, the step of estimating the probability that the subject is telling the truth or being deceptive includes multiple estimates obtained over a period of time during the subject's response, and the multiple estimates are weighted and combined to generate a binary output.

[0008] [Para 9] In other exemplary embodiments, the at least one video camera is capable of capturing frames at a rate of at least 100 frames per second, 50 frames per second, or 30 frames per second.

[0009] [Para 10] In another exemplary embodiment, the standoff device covers the entire face of the subject. A second video camera configured to record may be included. [Para 11] In another exemplary embodiment, the computing device comprises a stand-off It may be a cloud-based computing device located remotely from the device.

[0010] [Para 12] In another exemplary embodiment, the computing device comprises a standoff It may be part of the device or may be separate from the standoff device. [Para 13] In another exemplary embodiment, the subject is asked questions and the subject answers the questions. After allowing this, a period of time may be waited to re-establish a baseline state of ocular information for at least one eye of the subject before further questioning the subject.

[0011] [Para 14] In another exemplary embodiment, the entire utterance by the subject is recorded as an answer to the question. It can be evaluated as follows. [Para 15] In another exemplary embodiment, each binary output and at least one video camera The step of storing each corresponding video recorded by the camera can be performed by a computing device.

[0012] [Para 16] Eyeball system for assessing motion risk [Para 17] Exemplary embodiments of the present invention provide a method for evaluating a risk of motion based on ocular information of a subject. The method includes the steps of providing a video camera configured to record a close-up view of at least one eye of a subject, providing an electronic display screen configured to display a plurality of images to the subject, providing a computing device electronically connected to the video camera and the electronic display, displaying the at least one eye movement task on the electronic display, and The method includes recording eye information of at least one eye of the subject by a video camera during the task; processing the eye information by a computing device to identify changes in the subject's eye signals through the use of a convolutional neural network; evaluating by the computing device changes in the eye signals from the convolutional neural network combined with the at least one eye movement task corresponding to the changes in the eye signals by a machine learning algorithm; determining a job suitability result for the subject by the machine learning algorithm, the job suitability result being job suitable, job unsuitable, or more information needed; and displaying the job suitability result for the subject to the subject and / or a supervisor.

[0013] [Para 18] In another exemplary embodiment, the change in the eye signal is determined based on the following: eye movement, visual acuity, and / or visual acuity. Line position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, dilation duration The features may include any of: interval, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, and iris muscle changes.

[0014] [Para 19] In another exemplary embodiment, the at least one eye movement task comprises: The measurements may include any of the following: pupillary light reflex, optokinetic reflex, horizontal gaze nystagmus, smooth pursuit, gaze calibration, or startle response.

[0015] [Para 20] In another exemplary embodiment, the electronic display screen is a smartphone, tablet , a laptop screen, a desktop screen or an electronic screen. [Para 21] In another exemplary embodiment, a video camera, an electronic display screen, and a computer The computing devices can all be included as smartphones or tablets.

[0016] [Para 22] Exemplary embodiments of the present invention provide a method for evaluating a risk of motion based on ocular information of a subject. and providing a computing device electronically connected to the video camera and an electronic display; recording ocular information of at least one eye of the subject by the video camera; processing the ocular information by the computing device to identify changes in the subject's ocular signals through use of a convolutional neural network; evaluating by the computing device the changes in the ocular signals from the convolutional neural network by a machine learning algorithm; determining a fitness-for-job result for the subject by the machine learning algorithm, the fitness-for-job result being suitable, unsuitable, or more information needed; and displaying the fitness-for-job result for the subject to the subject and / or a supervisor.

[0017] [Para 23] In another exemplary embodiment, the change in the eye signal is determined based on the following: eye movement, visual acuity, and / or visual acuity. Line position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter) , acceleration (speed change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stromal change detection, eye region (strabismus), stromal deformation, iris muscle change.

[0018] [Para 24] In another exemplary embodiment, the fitness for work result may relate to the subject's level of intoxication. can. [Para 25] In another exemplary embodiment, the fitness for work result is related to the subject's impairment level. This can be done.

[0019] [Para 26] In another exemplary embodiment, the fitness for work result may relate to the subject's fatigue level. can. [Para 27] In another exemplary embodiment, the job fitness result is a subject's anxiety level and / or It can be related to stress levels.

[0020] [Para 28] Eyeball Systems for Optimizing Learning [Para 29] An exemplary embodiment of the present invention provides a method for optimizing learning based on subject's eye information. providing a computing device electronically connected to the video camera, the first electronic display, and the second electronic display; recording eye information of at least one eye of the subject by the video camera while learning the plurality of educational subjects; processing the eye information by the computing device to identify changes in the subject's eye signals through use of an optimization algorithm; providing a cognitive state model configured to determine a low to high cognitive load experienced by the subject, the cognitive load measuring an extent to which the subject utilizes mental resources; evaluating, by the computing device, the cognitive state model based on the changes in the eye signals to determine a probability of the subject experiencing a low to high cognitive load; and displaying, by the second electronic display, the probability of the subject experiencing a low to high cognitive load to the instructor.

[0021] [Para 30] In another exemplary embodiment, the method includes: The method includes the steps of establishing a position of a first electronic display in relation to at least one eye of the subject, determining a gaze position of the subject in relation to a plurality of educational topics from changes in eye signals, linking the gaze position of the subject and changes in eye signals for the plurality of educational topics to the subject's cognitive load, and displaying to the instructor via a second electronic display the subject's cognitive load in relation to the plurality of educational topics.

[0022] [Para 31] In another exemplary embodiment, the method performs a power spectral density frequency transform: Utilization may include separating a subject's pupil dilation resulting from changes in cognitive load from changes in surrounding luminance.

[0023] [Para 32] In another exemplary embodiment, the method includes a priori determination of a representative population or subjects. providing an optimal learning scale model having a learning scale of the subject based on the data, the learning scale ranging from understimulation to overstimulation; The method may include evaluating changes in eye signals by a visualizing device to determine the subject's position along the learning scale, and displaying the subject's position along the learning scale to the instructor by a second display.

[0024] [Para 33] In another exemplary embodiment, the method comprises: the step of: providing a memory formation model configured to determine a strength of the subject's short-term and / or long-term memory in relation to the plurality of educational topics; evaluating, by the computing device, changes in the eye signals to determine a strength of the subject's short-term and / or long-term memory in relation to the plurality of educational topics; and displaying, by the second display, to the instructor, the strength of the subject's short-term and / or long-term memory in relation to the plurality of educational topics.

[0025] [Para 34] In another exemplary embodiment, the change in the eye signal is determined based on the following: eye movement, visual acuity, and / or visual acuity. Line position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, dilation duration The features may include any of: interval, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, and iris muscle changes.

[0026] [Para 35] In another exemplary embodiment, a video camera is used while learning multiple educational subjects. The step of recording ocular information of at least one eye of the subject with a camera may also include recording facial expressions and / or posture of the subject with a camera while studying the plurality of educational subjects.

[0027] [Para 36] An exemplary embodiment of the present invention is a method for measuring cognitive load based on eye information of a subject. The method includes providing a video camera configured to record a close-up view of at least one eye of a subject, providing a computing device electronically connected to the video camera and an electronic display, recording eye information of at least one eye of the subject by the video camera, processing the eye information by the computing device to identify changes in the subject's eye signals through use of a convolutional neural network, evaluating by the computing device the changes in the eye signals from the convolutional neural network by a machine learning algorithm, determining the subject's cognitive load by the machine learning algorithm, and displaying the subject's cognitive load to the subject and / or a supervisor.

[0028] [Para 37] In another exemplary embodiment, the change in the eye signal is determined based on the following: eye movement, visual acuity, and / or visual acuity. Line position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (change in velocity), twitch (pupil change acceleration) , pupil fluctuation trace, convergence latency, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation , stromal change detection, eye area (strabismus), stroma deformation, and iris muscle changes.

[0029] [Para 38] Transillumination of the iris muscles to estimate stromal deformation [Para 39] An exemplary embodiment of the present invention is a method for detecting iris physiology and cognitive status of a subject. and / or an emotional state, the method comprising the steps of: providing a computing device; providing a video camera configured to record a close-up view of at least one eye of a subject; providing a first light configured to be held to skin of a lower eyelid of the subject, allowing light to shine from within the at least one eye; providing a second light located at a distance away from the subject and configured to be out of contact with the subject, the second light configured to illuminate the stroma of at least one eye of the subject, the first light and the second light both being electronically synchronized and configured to alternately flash; and engaging a user in a plurality of tasks, wherein each task of the plurality of tasks is The method includes the steps of: recording, with a video camera, ocular information including responses in the iris musculature and corresponding distortions in the stroma due to the cognitive and / or emotional states of the subject caused by the multiple tasks; processing, with a computing device, the ocular information through the use of an optimization algorithm to identify correlations between the responses in the iris musculature and the distortions in the stroma; and identifying, with the computing device, at least one predicted distortion in the stroma captureable using only the visible spectrum camera that correlates to a predicted response of the iris musculature when the subject was in the cognitive and / or emotional states caused by the multiple tasks.

[0030] [Para 40] In another exemplary embodiment, the first light is a (150mw) NIR LED The second light may include a (150mw) NIR LED.

[0031] [Para 41] In another exemplary embodiment, the first light and the second light are alternately ( It can be configured to blink at (160Hz) resulting in (80Hz) effect.

[0032] [Para 42] In another exemplary embodiment, the method includes generating a near-infrared image from a visible light image. The method may further include a method of performing a visual acuity assessment of at least one eye of a subject, the method including the steps of providing a visible spectrum video camera configured to record a close-up view of at least one eye of the subject, recording ocular information including distortions in the stroma due to the cognitive and / or emotional state of the subject with the visible spectrum video camera, and predicting, by a computing device, an infrared image of the at least one eye of the subject through a generative adversarial network using the ocular information from the visible spectrum video camera, wherein the prediction utilizes the at least one predicted distortion in the stroma to create the infrared image by the computing device.

[0033] [Para 43] Other features and advantages of the present invention may be seen in the following detailed description taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention. These will become apparent from the more detailed description below when taken together. [Para 44] The accompanying drawings illustrate the invention. [Brief description of the drawings]

[0034] [Figure 1] [Para 45] A front view of a subject's eye illustrating a masking technique based on the UNET neural network. [Figure 2A] [Para 46] FIG. 4 is a side view of the camera system of the present invention. [Figure 2B] [Para 47] FIG. 4 is a schematic top view of a subject utilizing the present invention. [Diagram 3] [Para 48] FIG. 4 is a flow chart of one embodiment of the present invention. [Figure 4] [Para 49] An example of the pupillary light reflex test of the present invention is shown. [Diagram 5] [Para 50] FIG. 1 is a diagram showing an example of a visual motor reflex according to the present invention. [Figure 6] [Para 51] A diagram showing an example of horizontal gaze nystagmus of the present invention. [Figure 7] [Para 52] A diagram showing an example of smooth pursuit of the present invention. [Figure 8] [Para 53] A diagram showing an example of gaze calibration of the present invention. [Figure 9] [Para 54] A diagram showing one embodiment of the software output of the present invention. [Figure 10] [Para 55] A diagram showing another embodiment of the software output of the present invention. [Figure 11] [Para 56] FIG. 1 is a diagram of one embodiment of cognitive load and learning parameter output of the present invention. [Figure 12] [Para 57] FIG. 1 illustrates one embodiment of trans-illumination hardware and process. [Figure 13] [Para 58] FIG. 13 is a still image of superficial stroma and transilluminated iris video captured using the transillumination hardware of FIG. [Figure 14] [Para 59] An example of GAN generation of iris IR images, where the generated IR image is used to create a CV mask, which is then projected onto the visible light image in real time. [Figure 15] [Para 60] An example of a CV mask formed on an IR image using UNET prediction and overlaid on a visible light image. [Figure 16] [Para 61] FIG. 1 is a perspective view of one embodiment of a dual camera design that simultaneously captures both NIR and visible light at the same location. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] [Para 62] In this specification, any reference to "Senseye" in this application is Please note that the references refer to the companies (i.e., applicants) involved. [Para 63] Eyeball System for Deception Detection: [Para 64] The Senseye deception detector uses eye signals to detect structured questions, A standoff device designed to detect deception in a wide variety of settings, including dynamic interrogation and passive observation of humans. The device records eye signals and classifies a person's statements as true or deceptive. The device provides binary classification. The classification of each question is based solely on eye information acquired during the response or statement, and thus the system design allows for individual classification of each question without requiring duplicate questions or specific question structures. This is an advancement over many deception detection systems and techniques that rely on multiple instances of the topic of the question to reach a conclusion or rely on comparing the results of questions with each other. The deception threshold can be set based on the use case (e.g., stricter parameters for higher stakes situations).

[0036] [Para 65] The deception detector uses a combination of models of cognitive and emotional states to Inputs are made into a deception model and classification. Thus, the system is capable of outputting a binary classification of the component model results. The system outputs a high or low cognitive load classification, which measures the degree to which a person utilizes mental resources to formulate their response. The system outputs a high or low arousal classification based on the subject's neural activation. Both of these measures are intended to provide a context for the deception classification.

[0037] [Para 66] Those skilled in the art will recognize that the Senseye deception detector is a stand-off device. It should also be understood that the subject's visual acuity may be reconfigured so as not to be visible and may instead reside, at least in part, in headgear, hats, glasses, etc. worn or held by the user. Monitoring and observing a subject in this manner is more intrusive, but still employs the remaining methods and strategies taught herein.

[0038] [Para 67] Deception detectors rely on eye signals to make classifications. Changes in these eye signals are: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), refixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency The features may include any of the following: blink period, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stromal change detection, eye region (strabismus), stroma deformation, and iris muscle changes.

[0039] [Para 68] The signal is generated by a multi-step process designed to extract subtle information from the eye. The methods are acquired using a process that is similar to that used in the previous work. As shown in Figure 1, image frames from the video data are processed through a series of optimization algorithms designed to isolate and quantify structures of interest. These isolated data are further processed using a mixture of automatically optimized, manually parameterized, and non-parametric transformations and algorithms. Leveraging the time series characteristics of these signals and cognitive load and arousal context information, some of these methods specifically estimate the probability that the input data represents a deceptive state. Multiple estimates are combined and weighted to generate a model that classifies responses or utterances based on the eye signals occurring during such responses or utterances as true or deceptive.

[0040] [Para 69] The result of computer vision segmentation according to the present invention The product works by processing eye metrics and analyzing these outputs during time-linked events occurring in the world. One version of the deception detector works in a specific Senseye hardware design. This embodiment of the device (see FIG. 2A) has a box with an aperture designed to be placed at eye level away from the participant. A video camera, such as a webcam, facing the participant captures the participant's head and parses the eyes using key points on the face. Once the eye position is identified, a high-resolution camera automatically adjusts to obtain a close-up view of the left eye (or right eye). As shown in the embodiment of FIG. 2A, there is a mirror 2A.1, a USB webcam 2A.2, a Canon 70-300USM AF lens 2A.3, a C-mount to Canon EF adapter 2A.4, an Emergent HR-12000-SM ​​2A.5, and a 10GigE SFP+ 2A.6.

[0041] [Para 70] camera enables ultra-high fidelity iris segmentation. The 500mm f / s camera allows update rates of over 100 frames per second, making even the most subtle and fastest changes in eye physiology detectable. However, slower frame rates such as 50 or 30 frames per second may be used. The adapter mount allows for a focal length that can fill the frame with the eye from over a meter away. In addition, the adapter allows the Senseye system to control the focus ring and aperture through software. The video data is stored in raw format, and processed tabular data is stored in a local database.

[0042] [Para 71] One possible use case showing the placement and distance of the user, subject and system. An example is also shown in Figure 2B. The camera is positioned perpendicular to the user and captures the eye using a 45 degree angle mirror. The user is asked to autofocus the lens on a session-by-session basis. The system avoids the need for manual focus, eliminating a point of human error. As shown in the embodiment of FIG. 2B, subject 2B.1 is at a distance 2B.2 of 36 inches from fixation point 2B.3. Camera 2B.4 is at a distance 2B.5 of 24 inches. As shown here, experimenter 2B.6 is at a distance 2B.7 of approximately 52 inches. Also shown in the table is device 2B.8 of the present invention.

[0043] [Para 72] This hardware setup is designed to make the system function. This is one of several ways in which the system can offload the computational workload to external solutions such as cloud instances or on-site compute nodes.

[0044] [Para 73] In both cases, the system works according to the outline in Figure 3. Starting the assessment At times, the system records metrics that produce a baseline reading for the subject. Many of the metrics listed above and included in the model are based on various types of variations from these baseline readings. These variations act as inputs to the model, allowing responses to be quickly classified as true or deceptive. Between each question, time is allowed for the interviewee's neurological and ocular metrics to return to baseline. After this reset period, usually 10 seconds, the interviewer proceeds to the next question, again receiving a quick classification of deceptive or true following the response.

[0045] [Para 74] In addition to quick classification, the system also provides a centralized system that displays the results given for each question. The system outputs a post-session report. The system provides an option to download a data file containing readings for each metric in the model timestamped for events occurring throughout the session. The system has the option to go back and view any previously recorded session video and classification results. The system has other features that make it flexible for a variety of use cases. The system provides an option to create templates of questions, which can be sequenced and automated for repeat screening. The system can also operate without templates for open questions. Finally, the system can make a video of the participant speaking with no questions being asked. In this case, the entire video utterance is considered as one question by the system, and once the video is completed, the classification is output in the same format, with the same post-action options.

[0046] [Para 75] Eyeball system for assessing motion risk: [Para 76] The Senseye Operation Risk Management (ORM) system is a system that monitors the suitability of workers for their tasks. The ORM system provides an objective measure of fitness. The system screens workers for excessive fatigue, impairment due to alcohol or drugs, and psychological risk factors that may interfere with job performance and safety. The system records video of the user's eyes while they perform various eye movement tasks and / or passively look at a screen. The ORM system also includes software that presents stimuli to the user. The system uses computer vision to segment the eye and quantify a wide range of ocular features. The eye metrics will then be input into a machine learning algorithm. The machine learning algorithm is designed to detect when a worker is too fatigued or impaired (due to drugs, alcohol or psychological risk factors) to safely perform their job. Fitness thresholds can be set based on the use case (e.g., stricter parameters for high stakes / high risk occupations). A further application of the ORM model and thresholds is that they can be implemented on videos that passively watch the user as they perform a task without the need for screening stimuli.

[0047] [Para 77] The primary input to the Senseye ORM system is the user's eyes. The system captures a video footage of a user's eyes while performing an eye movement task presented by a computer or passively looking at a phone or tablet screen or computer monitor. The location and identity of visible anatomical features from an open eye (i.e., sclera, iris, and pupil) are classified in the digital image pixel by pixel by a convolutional neural network originally developed for medical image segmentation. Based on the output of the convolutional neural network, a number of ocular features are generated. These eye metrics are combined with event data from the eye movement task that provides context and labels. The eye metrics and event data are provided to a machine learning algorithm, which returns a result of "fit for work", "not fit for work", or "more information needed". The system also returns the reason behind the "not fit for work" designation (e.g., excessive fatigue, suspected impairment from drugs or alcohol, excessive anxiety).

[0048] [Para 78] ORM relies on eye signals for classification. These eye signal changes are Below: Eye movement, Gaze position X, Gaze position Y, Saccade rate, Saccade peak velocity, Saccade average velocity, Saccade amplitude, Fixation duration, Fixation entropy (spatial), Gaze deviation (polar angle), Gaze deviation (eccentricity), Refixation, Smooth pursuit, Smooth pursuit duration, Smooth pursuit average velocity, Smooth pursuit amplitude, Scan path (gaze trajectory over time), Pupil diameter, Pupil area, Pupil symmetry, Velocity (change in pupil diameter), Acceleration (velocity change), Jerk (pupil change acceleration), Pupil fluctuation trace, Convergence latency , dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

[0049] [Para 79] The Senseye ORM system is available with a wide range of hardware options. The software is designed to run on a 32-bit or 64-bit processor. Eye videos can be captured by a webcam, a cell phone camera, or any other video camera with sufficient resolution and frame rate. Stimuli can be presented on a cell phone, tablet, or laptop screen or a standard computer monitor. The hardware required to run the software is a neural network-enabled FPGA, ASIC, or accelerated hardware located in the device or on a server accessed through an API.

[0050] [Para 80] Senseye ORM evaluation is based on the fact that users log in to the system. The process begins with initiating a user's eye movement task. This can be accomplished by typing a username and password, or by using facial recognition. In one embodiment, the user is presented with a series of eye movement tasks that can include pupillary light reflex, optokinetic reflex, nystagmus test, and smooth pursuit. Gaze calibration tasks can also be included to improve the gaze measurement output by the system. Each task is briefly described below. Depending on the use case, a subset of these tasks will be included. In another embodiment, the scan is designed to be more passive, so that the user's eyes are recorded while passively looking at the screen.

[0051] [Para 81] Figure 4 shows an example of the pupillary light reflex. The pupillary light reflex manipulates the brightness of a screen. The ORM system measures pupil size by visualizing the crosshairs in the center of the screen. The user fixates on a cross in the center of the screen while the screen changes from gray to black to white and back to black. The bright white screen causes the pupils to constrict. Pupil size is measured using computer vision and various parameters such as convergence latency, velocity and amplitude are calculated. Atypical pupil dynamics can be a symptom of fatigue, intoxication, stress, and other conditions. may show symptoms such as stress / anxiety, and sympathetic hyperactivity indicative of PTSD.

[0052] [Para 82] Figure 5 shows an example of a visual-motor reflex. The visual-motor reflex is moving across a screen. It is caused by presenting the user with alternating black and white bars. The eye reflexively tracks the bar as it moves across the screen, then moves back to the starting point as the bar moves away from the screen. This produces a characteristic sawtooth pattern in the eye's gaze x position, which should correspond to the stimulus velocity. Deviations from the stimulus velocity indicated that the visuomotor reflex was dysfunctional.

[0053] [Para 83] Figure 6 shows an example of horizontal gaze nystagmus. The nystagmus test is used by law enforcement agencies. The test is similar to components of field sobriety tests. A circular stimulus moves horizontally across the screen, traversing a 45 degree visual angle in each direction. The user is instructed to track the stimulus with their eyes. In healthy individuals, involuntary horizontal eye oscillations are expected to occur when the eyes move 40-45 degrees to the right or left. If the user is intoxicated, this involuntary movement will occur over a smaller visual angle.

[0054] [Para 84] Figure 7 shows an example of smooth pursuit. This task is performed while the user is moving at a constant speed. The test requires subjects to track a circular stimulus with their eyes. The ability to accurately track the stimulus is quantified in terms of speed and spatial precision. Poor alignment of speed or spatial position indicates impairment.

[0055] [Para 85] Figure 8 shows an example of gaze calibration. The gaze calibration task consisted of 11 The task consists of a series of dots displayed at different spatial locations. The user is asked to fixate each dot as it appears. This task is used to calibrate an eye-tracking system to provide accurate gaze information that is used to assess behavior in other tasks.

[0056] [Para 86] Startle response (not shown) was tested using a white noise pre-exposure. This is when the user can be tested with immeasurably large bursts of noise. Rapid and large dilation in response to noise bursts indicates sympathetic hyperactivity.

[0057] [Para 87] Continuation of the development of the ORM model based on the stimuli and metrics described above. is for use in passive surveillance situations. In these situations, the product does not act as a screening device, but rather outputs classification states from the model through video observation of a user performing a task. These models and thresholds utilize the same metrics listed above, but are less context dependent due to the transfer of learning from one scenario to another.

[0058] [Para 88] Eyeball Systems for Optimizing Learning: [Para 89] The Senseye Target Learning System (TLS) is a non-invasive eye measure of cognitive activity. The TLS algorithms monitor and classify cognitive events and states, including cognitive effort, short-term and long-term memory utilization and encoding, and arousal levels. These metrics serve individual purposes as indicators of the cognition required during a given task. Together, they can indicate when a person is in a state conducive to optimal learning. Over time, they can quantify a person's learning trajectory. Used in conjunction with a diverse learning curriculum, the TLS helps rapidly adapt the curriculum to an individual's unique learning pace. This level of adaptive training results in faster learning while ensuring retention of curriculum material. The targeted learning system includes outputs of cognitive load, Senseye Learning Parameters (SLP), and instances of short-term and long-term memory encoding.

[0059] [Para 90] TLS relies on eye signals for classification. These eye signal changes are Below: Eye movement, Gaze position X, Gaze position Y, Saccade rate, Saccade peak velocity, Saccade average velocity, Saccade amplitude, Fixation duration, Fixation entropy (spatial), Gaze deviation (polar angle), Gaze deviation (eccentricity), Refixation, Smooth pursuit, Smooth pursuit duration, Smooth pursuit average velocity, Smooth pursuit amplitude, Scan path (gaze trajectory over time), Pupil diameter, Pupil area, Pupil symmetry, Velocity (change in pupil diameter), Acceleration (velocity change), Jerk (pupil change acceleration), Pupil fluctuation trace, Convergence latency , dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

[0060] [Para 91] The signal is generated by a multi-step process designed to extract subtle information from the eye. Image frames from the video data are processed through a series of optimization algorithms designed to isolate and quantify structures of interest. These isolated data are further processed using a mixture of automatically optimized, manually parameterized, and non-parametric transforms and algorithms.

[0061] [Para 92] Cognitive load: [Para 93] The TLS software runs on any device with a front-facing camera (tablets, The TLS software can function on any device (phone, computer, VR headset, etc.) using anatomical signals (more specifically, physiological signals) extracted from the images to predict various cognitive states through optimization algorithms. The algorithms provide an estimated probability that the input data represents a particular cognitive state and can identify the presence of one or more cognitive states. The image signals are run through a series of data processing operations to extract signals and estimates. First, multiple image masks are applied to separate ocular components and facial features, allowing various metrics to be extracted from the images in real time. From the image filters, relevant signals are extracted through transformation algorithms that support a final estimate of the cognitive state. Multiple data streams and estimates can be made in a single calculation and the cognitive load signal can result from a combination of multiple unique processing and estimation algorithms. The cognitive load output can be directly linked to the stimuli (video and / or image and / or blank screen shown) by relating the event time and time course of the cognitive load output. The software can display, in real time, the cognitive load of the individual as the event is occurring (Figures 9 and 10).

[0062] [Para 94] TLS products use various gaze patterns to perform inferences about cognitive states. It is also possible to utilize the gaze estimation. The gaze used in this product falls into three main categories: 1) eye center estimation in the frame, 2) eye position and orientation estimation, and 3) 3D viewpoint estimation of the subject's focus in space. The information distilled from all these techniques can be used individually or in concert. Individually, these methods provide uniquely useful measurements of eye movements, which together (with or without additional calibration routines) provide cascading information parameters that are used to build a 3D model of the eye and gaze vector. The gaze point on an object in real space, such as a computer monitor, can then be estimated by intersecting the gaze vector with a corresponding 2D plane parallel to the object's surface. The monitor, IR lights, nIR lights and camera position are all known quantities prior to gaze estimation. The participant's gaze is projected in the form of a heat map onto the screen the participant is looking at. By plotting the cognitive load at the time of gaze, the software links the gaze position and the cognitive load associated with gaze. This allows individuals to analyze exactly which location / object / task a participant was looking at when there was a change in their individual cognitive load output.

[0063] [Para 95] It is unlikely that the stimuli a user sees will have a constant luminance. It is well known that perceptual changes due to luminance are the main drivers of pupillary responses. To account for luminance-based pupillary responses, TLS uses a power spectral density (PSD) frequency transform to separate the resulting pupil dilation from cognitive load. The PSD transform measures the power of the waveform at each specific frequency within an interval. Using this method, the various types of sine waves that make up any kind of wave can be determined. Decomposition of pupillary waveforms through the PSD has been shown to detect cognitive load independent of luminance conditions (Marshall, 2002; Nakayama & Shimizu, 2004; Hampson et al., 2010; Peysakhovich et al., 2015; Peysakhovich et al., 2017; Reiner & Gelfeld, 2014). Luminance responses are reflexive and fast, whereas pupil changes due to cognitive processes are slower (Joshi et al., 2016). Using a mixture of measured luminance and pupil response signals, the TLS algorithm applies PSD and other transformations to create a new combined signal derived from multiple time and frequency signals. These signals then drive probability estimates of cognitive states through an optimization algorithm to identify cognitive load states even in the presence of pupil responses from external light sources.

[0064] [Para 96] Senseye learning parameters: [Para 97] As part of the TLS, Senseye uses the Senseye learning parameters (S We developed a TLS algorithm (SLP) that considers the individual's internal factors and represents them as a scale from low engagement / understimulation to high internal state / overstimulation. This is calculated using an algorithm that converts the individual's eye signals into a reading on the optimal learning scale. The optimal learning scale is statistically based on representative population or individual prior data (see Adaptive Senseye Learning Parameters). When the participant's internal state is low (minimal cognitive load is maintained), the indicator shifts to the low side of the SLP scale, whereas a high internal state (high cognitive load is maintained and indicated as overstimulation) shifts the SLP indicator to the high side of the scale. This allows the instructor to adapt and adjust the tasks to allow the participant to remain at the optimal learning point (middle of the SLP) for the best learning outcome (Figure 11).

[0065] [Para 98] Adaptive Senseye learning parameters: [Para 99] As explained above, SLP operates on fixed equations and optimizes Learning parameters can be generated. However, SLP also has the ability to change parameters according to the subject's expertise and learning ability. The amount of stress that an individual can undergo while still absorbing new information varies from person to person. Under the same amount of stress and arousal, some people maintain their learning ability while others do not. Variations in cognitive performance at various arousal levels have been observed in previous studies (Chaby et al., 2015; Yerkes and Dodson, 1908; Anderson, 1994). SLP's adaptive function uses the ability to judge an individual's expertise (beginner, intermediate, advanced) and correlates the performance with cognitive load to automatically generate an optimal scale for the individual. The scale can shift and adjust based on the changes in the individual's performance and cognitive load under stress conditions as the individual learns and masters the task. This function further improves the customizability of quantified learning, allowing instructors or automated lesson systems to more efficiently change the curriculum according to individual learning profiles.

[0066] [Para 100] Memory classification: [Para 101] TLS can also differentiate the occurrence and strength of memory formation, including but not limited to the formation of short-term memory (STM) and long-term memory (LTM) during the learning process. Prior literature has shown that different brain regions are involved in various types of memory formation. The frontal cortex is associated with LTM, while the hippocampus is closely associated with STM. Individuals with lesions or damage to the frontal cortex often have difficulty encoding and retrieving memories (Jetter et al., 1986; McAndrews and Milner, 1991; Eslinger and Grattan, 1994; Stuss et al., 1994; Moscovitch and Winocur, 1995), whereas they show little or no impairment in short-term memory (Kesner et al., 1994; Stuss et al., 1994; Swick and Knight, 1996; Dimitrov et al., 1999; Alexander et al., 2003). The hippocampus is known to be involved in the formation of short-term memory, and lesions of the hippocampus impair the encoding of new memories ( Jonides et al., 2008 ; Cohen and Eichenbaum, 1993 ).

[0067] [Para 102] The frontal cortex is not only involved in LTM, but is also important in generating various eye movements through transmitting motor commands to the brainstem. It is also known to regulate the changes in pupil diameter (Schlag-Ray et al., 1992; Ebitz and Moore, 2017) associated with memory formation and retrieval (Kucewicz et al., 2018). Because both LTM and eye metrics are related to the frontal cortex, eye metrics can be utilized to read out memory formation and build a model based on the various patterns of eye metrics that occur during the formation of LTM and STM. Using this model, TLS built a function that outputs the strength and type of memory formation that occurs while a person is engaged in a learning task.

[0068] [Para 103] Figures 9 and 10 show one embodiment of the TLS software output of a cognitive load session. These figures represent iterations of the TLS user interface. Figure 9 is a live time series analysis of the user's cognitive load and SLP output during a particular session. In Figure 10, there is an instantaneous feed of both the cognitive load and the SLP, as well as a live feed of the eye detection and masking algorithm used to output the respective features. The eye detection algorithm locates both eyes on the user's face and focuses on one eye to segment the iris, pupil and sclera. The instantaneous readout of the cognitive load is reported visually in the form of a gauge and quantitatively in the form of a percentage or similar interpretation of the user's cognitive load.

[0069] [Para 104] Figure 11 is one embodiment of the UI / UX of TLS cognitive load and senseye learning parameters output. One representation of cognitive load level and learning state along with SLP. The grey dot is the participant's current state. At the top is the cognitive load meter, where low cognitive load is represented by green areas and high cognitive load areas are presented in red. At the bottom is the SLP. As with the cognitive load readout, the grey dot represents the participant's current learning state. The ends of the scale represent suboptimal learning states due to under-stimulation and over-stimulation.

[0070] [Para 105] Transillumination of the iris muscles to estimate stromal deformation: [Para 106] As a general overview, previous literature has demonstrated that the iris muscles of the eye are innervated by specific regions of the brain. Activating these brain regions results in complementary changes within the respective muscles of the iris, leading to the hypothesis that iris physiology may provide a non-invasive means to quantify relevant cognitive states. Notably, direct observation of iris muscle physiology is obscured by the overlying membrane, known as the stroma. The technique outlined here, hereafter referred to as "transillumination," allows visualization of iris muscle physiology and anatomy and then allows these patterns of muscle movement to be interpreted as distortions within the overlying stroma of the iris. Transillumination is a method for visualizing how iris muscle physiology manifests itself. By mapping the association between known muscle physiology and known patterns of stromal strain, transillumination allows users to infer complex patterns of iris muscle physiology from simple surface-level video recordings of the eye. Transillumination is an essential technique for accessing brain signals from the eye.

[0071] [Para 107] Senseye has developed a technique to analyze the contribution of individual muscle groups and fibers to iris movements associated with cognitive and emotional states, and to map these movements onto the surface layer of the eye, the stroma, which is visible to a commercially available camera. This innovation is a novel and major step towards achieving a contact-free way of reading brain activity from ocular metrics. It involves both conceptual and technological innovations. The conceptual innovation is in examining the individual movements of muscles under various cognitive states to extract reliable signals. The technological innovation is the way in which the stroma and transilluminated muscles are visualized such that they can be mapped onto each other.

[0072] [Para 108] The muscles of the iris are innervated by the parasympathetic and sympathetic nervous systems. In particular, the dilator muscles of the iris are innervated by many individual endings of the SNS, and the sphincter muscles are innervated by many individual endings of the PNS. These innervations allow information to travel downstream along these neural pathways to the individual muscles of the iris, resulting in movement that can be measured and used to infer cognitive and emotional states. Transillumination techniques that view the iris muscles and map them onto the stroma allow for the creation of Senseye products that use surface-level changes in the iris to model brain activity.

[0073] [Para 109] Regarding the process, the signal acquisition device consists of two lighting components, henceforth referred to as "Lighting Component 1" (LC1) and "Lighting Component 2" (LC2), and one camera. LC1 is a single 150mw nIR LED powered at 5 volts. It is held to the skin of the lower eyelid so that the light shines from within the iris musculature, making the iris musculature visible. LC2 is a 150mw nIR standoff LED array that illuminates the outer stroma of the iris (Figure 1). These lighting systems are synchronized to alternately flash at 160hz using an oscilloscope, producing the effect of two perfectly synchronized 80hz videos, one of the iris musculature, and one of the outer stroma (Figure 2). Video data is collected using the camera at a frame rate and resolution capable of capturing the fine movements of the iris muscles.

[0074] [Para 110] The data collection protocol places the participant in a seat in front of the camera while a set of automated instructions and tasks are presented (Figure 12). The application has configurable timing for each task and changes the image on the screen in front of the participant to induce eye responses, providing automated instructions on when to perform specific behaviors that evoke cognitions or emotions, such as doing arithmetic or placing hands in ice. The application outputs specific time-stamped events in the form of tabular data, allowing the recorded film footage to be synchronized with the generated stimuli for appropriate analysis.

[0075] [Para 111] In the next set of analyses, image frames from the video data are processed through a series of optimization algorithms and transformations designed to isolate and quantify structures of interest. Data derived from images illuminated by LC1 are used to parse structures from direct observation of the iris musculature. Data derived from images illuminated by LC2 are used to parse distortions within the stroma overlying the iris. The resulting image pairs provide a unique surface-to-subsurface mapping of involuntary iris muscle motion. Signals extracted from these images are collected in a structured format and stored with associated experimental metadata capable of contextualizing a wide range of cognitive states and processes. These novel datasets are used to parse brain activity and superficial stroma motion in a measurable and reliable manner. can be directly mapped to the subsurface iris activity with the formula:

[0076] [Para 112] Figure 12 shows one embodiment of the trans-illumination hardware and process. An LED is held at eye position while the participant is viewed being illuminated at 160hz alternating by an LED on a monitor. A participant-facing camera captures video of the eye.

[0077] [Para 113] Figure 13 is a still image of the superficial stroma and transilluminated iris video captured using transillumination hardware. 13.1 refers to the transilluminated sphincter, 13.2 refers to the transilluminated dilator, and 13.3 refers to the superficial stroma.

[0078] [Para 114] Method for generating NIR images from an RGB camera [Para 115] The method of the present invention uses a generative adversarial network and a combination of visible and IR light, which is discussed further herein. Continuing the theme of creating a mapping between subsurface iris structures visible in IR light and surface structures visible in visible light, Senseye has developed a method to project an iris mask formed on an IR image for data extracted from visible light. The technique uses a generative adversarial network (GAN) to predict an IR image of an input image captured under visible light (see FIG. 14). A CV mask is then run on the predicted IR image and overlaid back onto the visible light image (see FIG. 15).

[0079] [Para 116] Part of the method is generating a training set of images from which the GAN learns to predict IR images from visible light images (see Figure 14). Senseye developed a hardware system and experimental protocol to generate these images. The setup consists of two cameras, one color-sensitive and one NIR-sensitive (see items 16.1 and 16.2 in Figure 16). The two are placed tangent to each other, with the hot mirror at a 45 degree angle to both (see item 16.3 in Figure 16). The center of gravity of the first surface of the mirror is equidistant from both sensors. Visible light passes in a straight line through the hot mirror to the visible sensor, and NIR bounces into the NIR sensor. Thus, the system creates highly optically aligned NIR and color images that can be overlaid pixel by pixel. A hardware trigger is used to ensure that the cameras are exposed simultaneously with less than 1 μS error.

[0080] [Para 117] Figure 16 is a diagram of a hardware design that captures NIR and visible light video simultaneously. Two cameras, one with a near-IR sensor and one with a visible light sensor, are mounted on a chassis at a 45 degree angle with a hot mirror (invisible to one camera sensor, an opaque mirror to the other) to create an image overlay with pixel-level accuracy.

[0081] [Para 118] Creating optically and temporally aligned visible and NIR datasets with low error allows Senseye to create large, diverse datasets that do not require labeling. Alignment, rather than manual labeling, allows Senseye to use NIR images as a reference to train color images. Existing networks already have the ability to classify and segment the eye into sclera, iris, pupil, etc., giving the ability to use their outputs as training labels. Additionally, unsupervised techniques such as pix-to-pixGAN leverage this framework to model similarities and differences between image types. These data are used to create surface-to-surface and / or surface-to-subsurface mappings of visible and invisible iris features.

[0082] [Para 119] Another possible way to properly filter the RGB spectrum so that it resembles the NIR image is to make the rendered image appear in natural and NIR light. The goal of this study is to use a simulation of the eye to resemble both natural light in the IR light spectrum. The neural network architecture is similar to those listed above (pix-to-pix), and the goal is to allow the subcorneal structures (iris and pupil) to be properly reconstructed and segmented despite reflections or other artifacts caused by the interaction of the natural light spectrum (360-730 nm) with the particular eye.

[0083] [Para 120] The utility of GANs is that they learn functions capable of generating NIR images from RGB images. The problem with RGB images comes from the degradation of contrast between the pupil and the iris, especially for darker eyes. What this means is that if the eye is not flooded with enough light, the border between the brown iris and the pupil hole is indistinguishable due to the proximity of the color spectrum. In RGB space, due to not controlling the specific spectrum of light, we are at the mercy of another property of the eye, which acts as a mirror. This property allows any object to appear as a transparent film over the pupil / iris. An example of this is the possibility of creating a smaller version of a bright monitor on the eye given an rgb image. For this reason, the GAN acts as a filter. The GAN is able to filter out reflections, sharpen the boundaries, and recover the true boundaries of the iris and pupil due to the learned embeddings.

[0084] [Para 121] While several embodiments have been described in detail for purposes of illustration, various modifications may be made thereto without departing from the scope and spirit of the invention. Accordingly, the invention is not to be limited except as by the appended claims.

Claims

1. 1. A method for detecting deception based on eye information of a subject, the method comprising: providing a stand-off device configured to view the subject during an examination, the stand-off device being out of physical contact with the subject, the stand-off device having at least one video camera configured to record a close-up view of at least one eye of the subject, the stand-off device having or being connected to a computing device; providing a cognitive state model configured to determine a high to low cognitive load experienced by the subject, the cognitive load measuring the extent to which the subject utilizes mental resources to formulate their response; providing an emotional state model configured to determine a high to low state of arousal experienced by the subject, the state of arousal being based on activation of the subject's nervous system; recording, with the at least one video camera, the ocular information of the at least one eye of the subject; establishing a baseline state of the ocular information of the at least one eye of the subject prior to questioning the subject; asking the subject questions and allowing the subject to answer the questions; processing the eye information after administering the questions and including time for the subject to respond to the questions to identify changes in the subject's eye signals; evaluating, by the computing device, the cognitive state model and the emotional state model based solely on the changes in the eye signals to estimate a probability that the subject is telling the truth or being deceptive; determining a binary output of truth or deception; displaying said binary output to an administrator; A method for detecting deception comprising:

2. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency.

2. The method of claim 1, comprising any of the following: blink period, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

3. 2. The method of claim 1, wherein the step of estimating the probability that the subject is telling the truth or being deceptive includes multiple estimates obtained over a period of time during the subject's response, the multiple estimates being weighted and combined to generate the binary output.

4. The method of claim 1 , wherein the at least one video camera captures frames at a rate of at least 100 frames per second.

5. The method of claim 1 , wherein the at least one video camera captures frames at a rate of at least 50 frames per second.

6. The method of claim 1 , wherein the at least one video camera captures frames at a rate of at least 30 frames per second.

7. The method of claim 1 , wherein the standoff device includes a second video camera configured to record the subject's entire face.

8. The method of claim 1 , wherein the computing device is a cloud-based computing device located remotely from the standoff device.

9. The method of claim 1 , wherein the computing device is part of the standoff device.

10. The method of claim 1 , wherein the computing device is separate from the standoff device.

11. 2. The method of claim 1, further comprising: after asking the subject the question and allowing the subject to answer the question, waiting a period of time before asking the subject any further questions and re-establishing the baseline state of the eye information for the at least one eye of the subject.

12. The method of claim 1 , wherein an entire utterance by the subject is evaluated as an answer to the question.

13. The method of claim 1 , further comprising storing, by the computing device, each binary output and each corresponding video recorded by the at least one video camera.

14. 1. A method for detecting deception based on eye information of a subject, the method comprising: providing a stand-off device configured to view the subject during an examination, the stand-off device being out of physical contact with the subject, the stand-off device having at least one video camera configured to record a close-up view of at least one eye of the subject, the stand-off device having or being connected to a computing device; providing a cognitive state model configured to determine a high to low cognitive load experienced by the subject, the cognitive load measuring the extent to which the subject utilizes mental resources to formulate their response; providing an emotional state model configured to determine a high to low state of arousal experienced by the subject, the state of arousal being based on activation of the subject's nervous system; recording, with the at least one video camera, the ocular information of the at least one eye of the subject; establishing a baseline state of the ocular information of the at least one eye of the subject prior to questioning the subject; asking the subject questions and allowing the subject to answer the questions; processing the eye information after administering the questions and including time for the subject to respond to the questions to identify changes in the subject's eye signals; evaluating, by the computing device, the cognitive state model and the emotional state model based solely on the changes in the eye signals to estimate a probability that the subject is being truthful or deceptive; determining a binary output of truth or deception; displaying said binary output to an administrator; Including, 1. A method of deception detection, wherein the step of estimating the probability that the subject is telling the truth or being deceptive includes a plurality of estimates obtained over a period during the subject's response, the plurality of estimates being weighted and combined to generate the binary output.

15. 1. A method for detecting deception based on eye information of a subject, the method comprising: providing a stand-off device configured to view the subject during an examination, the stand-off device being out of physical contact with the subject, the stand-off device having at least one video camera configured to record a close-up view of at least one eye of the subject, the stand-off device having or being connected to a computing device; providing a cognitive state model configured to determine a high to low cognitive load experienced by the subject, the cognitive load measuring the extent to which the subject utilizes mental resources to formulate their response; providing an emotional state model configured to determine a high to low state of arousal experienced by the subject, the state of arousal being based on activation of the subject's nervous system; recording, with the at least one video camera, the ocular information of the at least one eye of the subject; establishing a baseline state of the ocular information of the at least one eye of the subject prior to questioning the subject; asking the subject questions and allowing the subject to answer the questions; processing the eye information after administering the questions and including time for the subject to respond to the questions to identify changes in the subject's eye signals; evaluating, by the computing device, the cognitive state model and the emotional state model based solely on the changes in the eye signals to estimate a probability that the subject is telling the truth or being deceptive; determining a binary output of truth or deception; displaying said binary output to an administrator; storing, by the computing device, each binary output and each corresponding video recorded by the at least one video camera; A method for detecting deception comprising:

16. A method for assessing a risk of movement based on eye information of a subject, the method comprising: providing a video camera configured to record a close-up view of at least one eye of the subject; providing an electronic display screen configured to display a plurality of images to a subject; providing a computing device electronically connected to said video camera and to an electronic display; displaying at least one eye movement task on the electronic display; recording, with the video camera, the ocular information of the at least one eye of the subject during the at least one eye movement task; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of a convolutional neural network; evaluating, by the computing device, changes in the eye signals from the convolutional neural network combined with the at least one eye movement task corresponding to the changes in the eye signals using a machine learning algorithm; A step of determining a job aptitude result of the subject by the machine learning algorithm, the job suitability result being job suitable, job unsuitable, or more information needed; displaying the job fitness results of the subject to the subject and / or a supervisor; Methods for assessing operational risk, including:

17. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), refixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, 17. The method of claim 16, comprising any of the following: dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle changes.

18. 20. The method of claim 17, wherein the at least one eye movement task comprises any of the following: pupillary light reflex, optokinetic reflex, horizontal gaze nystagmus, smooth pursuit, gaze calibration, or startle response.

19. 20. The method of claim 17, wherein the electronic display screen is of a smart phone, a tablet, a laptop screen, a desktop screen or an electronic screen.

20. 20. The method of claim 17, wherein the video camera, the electronic display screen and the computing device are all comprised as a smart phone or a tablet.

21. A method for assessing a risk of movement based on eye information of a subject, the method comprising: providing a video camera configured to passively record a close-up view of at least one eye of the subject; providing a computing device electronically connected to said video camera and to an electronic display; recording, by the video camera, the ocular information of the at least one eye of the subject; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of a convolutional neural network; evaluating, by the computing device, a change in the ocular signal from the convolutional neural network using a machine learning algorithm; A step of determining a job aptitude result of the subject by the machine learning algorithm, the job suitability result being job suitable, job unsuitable, or more information needed; displaying the job fitness results of the subject to the subject and / or a supervisor; Methods for assessing operational risk, including:

22. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), refixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, 7. The method of claim 6, including any of the following: dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

23. 23. The method of claim 22, wherein the fitness-for-job result relates to the subject's level of intoxication.

24. 23. The method of claim 22, wherein the fitness-for-job result relates to a degree of functional impairment of the subject.

25. The method for assessing performance risk according to claim 22, wherein the job fitness result relates to a fatigue level of the subject.

26. 23. The method of claim 22, wherein the job fitness result relates to the subject's anxiety level and / or stress level.

27. A method for assessing a mental state of a subject based on eye information of the subject, the method comprising: providing a video camera configured to passively record a close-up view of at least one eye of the subject; providing a computing device electronically connected to said video camera and to an electronic display; recording, by the video camera, the ocular information of the at least one eye of the subject; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of a convolutional neural network; evaluating, by the computing device, a change in the ocular signal from the convolutional neural network using a machine learning algorithm; determining the mood of the subject with the machine learning algorithm; displaying the subject's mood to the subject and / or a supervisor; A method for assessing mood, including:

28. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, 28. The method of claim 27, comprising any of the following: smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (eye gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (acceleration of pupil change), pupil fluctuation trace, convergence latency, dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stromal change detection, eye area (strabismus), stromal deformation, and iris muscle change.

29. 29. A method for assessing mood as claimed in claim 28, wherein said mood relates to the subject's level of intoxication.

30. 29. The method of claim 28, wherein the mood relates to a degree of functional impairment of the subject.

31. 29. The method of claim 28, wherein the mood relates to the subject's degree of fatigue.

32. 29. A method for assessing a mood state according to claim 28, wherein the mood state relates to the subject's level of anxiety and / or stress.

33. A method for optimizing learning based on eye information of a subject, the method comprising: providing a video camera configured to record a close-up view of at least one eye of the subject; providing a first electronic display configured to display a plurality of educational subjects to the subject; providing a second electronic display configured to display the output to an instructor; providing a computing device electronically connected to the video camera, the first electronic display, and the second electronic display; recording, with the video camera, the ocular information of the at least one eye of the subject while the subject is learning the plurality of educational subjects; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of an optimization algorithm; providing a cognitive state model configured to determine a low to high cognitive load experienced by the subject, the cognitive load measuring the extent to which the subject utilizes mental resources; evaluating, by the computing device, the cognitive state model based on the changes in the ocular signals to determine a probability of the subject experiencing low to high cognitive load; displaying to the instructor, via the second electronic display, the probability of the subject experiencing low to high cognitive load; How to optimize learning, including:

34. Establishing a position of the first electronic display in relation to the at least one eye of the subject by the computing device; determining a gaze position of the subject in relation to the plurality of educational subjects from the changes in the eye signals; linking the gaze position of the subject and the changes in the eye signals of the plurality of educational subjects to a cognitive load of the subject; and providing the instructor with a second electronic display. displaying the subject's cognitive load in relation to the plurality of educational topics; 34. The method of optimizing learning according to claim 33, comprising:

35. 34. A method for optimizing learning as claimed in claim 33, comprising the step of separating the subject's pupil dilation resulting from changes in cognitive load from changes in surrounding luminance by utilising a power spectral density frequency transform.

36. 34. The method of optimizing learning of claim 33, comprising the steps of: providing an optimal learning scale model having a learning scale for the subject based on prior data for a representative population or subject, the learning scale ranging from understimulation to overstimulation; evaluating changes in the eye signals by the computing device to determine the subject's position along the learning scale; and displaying the subject's position along the learning scale to the instructor by the second display.

37. 34. The method of optimizing learning of claim 33, comprising the steps of: providing a memory formation model configured to determine a strength of short-term and / or long-term memory; evaluating, by the computing device, changes in the eye signals to determine the strength of the short-term and / or long-term memory of the subject in relation to the plurality of educational topics; and displaying to the instructor by the second display the strength of the short-term and / or long-term memory of the subject in relation to the plurality of educational topics.

38. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, and convergence latency. , dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

39. 34. The method of optimizing learning according to claim 33, wherein the step of recording eye information of at least one eye of the subject with a video camera while the subject is learning the plurality of educational subjects also includes recording facial expressions and / or posture of the subject with the camera while the subject is learning the plurality of educational subjects.

40. A method for measuring a cognitive load based on eye information of a subject, the method comprising: providing a video camera configured to record a close-up view of at least one eye of the subject; providing a computing device electronically connected to said video camera and to an electronic display; recording, by the video camera, the ocular information of the at least one eye of the subject; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of a convolutional neural network; evaluating, by the computing device, a change in the ocular signal from the convolutional neural network using a machine learning algorithm; determining the cognitive load of the subject with the machine learning algorithm; displaying the cognitive load of the subject to the subject and / or a supervisor; How to measure cognitive load, including:

41. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, and convergence latency. , dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, fractional blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye region (strabismus), stroma deformation, iris muscle change.

42. 1. A method for measuring short-term and / or long-term memory encoding based on ocular information of a subject, the method comprising: providing a video camera configured to record a close-up view of at least one eye of the subject; providing a computing device electronically connected to said video camera and to an electronic display; recording, by the video camera, the ocular information of the at least one eye of the subject; processing, by the computing device, the ocular information to identify changes in the subject's ocular signals through the use of a convolutional neural network; evaluating, by the computing device, a change in the ocular signal from the convolutional neural network using a machine learning algorithm; determining the cognitive load of the subject with the machine learning algorithm; displaying the cognitive load of the subject to the subject and / or a supervisor; A method for measuring short-term and / or long-term memory encoding, comprising:

43. The changes in the eye signals include the following: eye movement, gaze position X, gaze position Y, saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (velocity change), twitch (pupil change acceleration), pupil fluctuation trace, convergence latency, dilation duration. , spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contraction, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centering vector, blink rate, blink duration, blink latency, blink velocity, partial blink, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, eye area (strabismus), stroma deformation, iris muscle changes.

44. 1. A method for discovering a relationship between iris physiology and a cognitive and / or emotional state of a subject, said method comprising: Providing a computing device; providing a video camera configured to record a close-up view of at least one eye of the subject; providing a first light configured to be held to skin of a lower eyelid of the subject and allowing light to shine from within the at least one eye; providing a second light positioned at a distance away from the subject and configured to not contact the subject, the second light configured to illuminate a stroma of the at least one eye of the subject; the first light and the second light are both electronically synchronized and configured to alternately flash; engaging the user in a plurality of tasks, each task in the plurality of tasks configured to evoke a cognition or an emotion; recording, with the video camera, ocular information including responses in the iris musculature and corresponding distortions in the stroma due to the cognitive and / or emotional states of the subject caused by the plurality of tasks; processing, by the computing device, the ocular information through the use of an optimization algorithm to identify correlations between the response in the iris musculature and the distortion in the stroma; identifying, with the computing device, at least one predicted distortion in the stroma captureable using only a visible spectrum camera that correlates to a predicted response of the iris musculature when the subject is in the cognitive and / or emotional states produced by the plurality of tasks; Including,how to discover relationships.

45. 45. The method of claim 44, wherein the first light comprises a NIR LED.

46. 45. The method of claim 44, wherein the second light includes a NIR LED.

47. 45. The method of claim 44, wherein the first light and the second light are configured to alternately flash.

48. 45. The method of claim 44, wherein the first light includes a 150mw NIR LED.

49. 45. The method of claim 44, wherein the second light includes a 150mw NIR LED.

50. 45. The method of claim 44, wherein the first light and the second light are configured to flash alternately at 160 Hz, resulting in an 80 Hz effect.

51. The method further includes a method for generating a near-infrared image from a visible light image, the method comprising: providing a visible spectrum video camera configured to record the close-up view of the at least one eye of the subject; recording, with the visible spectrum video camera, the ocular information including the distortions in the stroma due to the cognitive and / or emotional state of the subject; predicting, by the computing device, an infrared image of the at least one eye of the subject through a generative adversarial network using the eye information from the visible spectrum video camera; Including, The method of claim 1 , wherein the prediction, by the computing device, utilizes the at least one predicted distortion in the stroma to generate the infrared image.

Images