Remote eye gaze cursor control technology

Abstract

A system adapted for interacting with a user from monitoring the activity of the user's eye gaze, the system comprising: a user webcam configured to determine a location of a user's eye gaze on a user display configured to display digital content to the user; an operator display configured to see the live interaction of the user's eye gaze from movement of the eye gaze controlled cursor projected in the operator display; a processor with a gaze estimation module for receiving eye-motion and head-motion signals from the user webcam as calibration data to use to estimate eye gaze, the processor configured to: generating an interactive stimulus to the user display; interacting with the interactive stimulus via user gaze-based interaction or blink-based interaction; and analysing and determining user focus based on eye tracking from interaction of the interactive stimulus.

Description

REMOTE EYE GAZE CURSOR CONTROL TECHNOLOGYTECHNICAL FIELD

[0001] The present invention or disclosure generally relates to eye tracking and, more specifically, to systems and methods of facilitating remote eye tracking control used as part of various audio-visual virtual conferencing platforms including those used for telehealth.BACKGROUND

[0002] Individuals with neurodisabilities may have limited control over movements and speech and rely only on their eye gaze to communicate. Due to in-person activities such as rehabilitation sessions being moved online due to the CO VID- 19 pandemic, such individuals have experienced reduced quality of learning, and health and social care outcomes. Without access to technological advancement including special tools to aid the experience of using computers and telehealth, clinicians working with these individuals are facing challenges in interaction and engagement during online sessions as well as providing the adequate support that in-person sessions bring.

[0003] The application of information and communication technologies in the field of rehabilitation is known as telerehabilitation. This telecommunication-based practice can provide remote support, assessment, and information to individuals including those with neurodisabilities without them having to leave their homes. Previous reports have shown that telerehabilitation improves the well-being of neuro-disabled individuals due to its ability to increase access to rehabilitation services and healthcare professionals. This impact is especially significant in under-serviced areas with limited access to these support resources. Thus, the challenge is two-fold: 1) providing access to health and social care via telerehabilitation and 2) making this tool effective in supporting individuals including those with neurodisabilities.

[0004] Current technological advancements have adapted the use of eye gaze control to allow more access to health and social care for individuals with speech impaired disabilities. A number of eye gaze devices are readily available, including TobiiDynavox portable eye trackers and eye gaze tablets. Studies suggest that these specialised eye gaze tracking devices enhance engagement and improve early communication skills in children with speech impaired neurodisabilities. However, they are inaccessible due to their high cost and inability to integrate with teleconferencing platforms. Additionally, there is currently limited opportunity for clinicians and educators to remotely supervise and assess eye gaze as a communication aided technology. The invention described herein attempts to address this gap of remote eye gaze control technology in tele-conferencing platforms and improved communication, learning and support between clinician, researchers and educators and neuro-disabled individuals which can be used when in-person rehabilitation sessions cannot be conducted. It bears mention that this invention is not limited nor conditional upon use by only neuro-disabled individuals and others can use it to the same or similar benefit.

[0005] Any discussion of the prior art throughout the specification should in no way be considered as an admission that such prior art is widely known or forms part of common general knowledge in the field.SUMMARY OF INVENTION

[0006] PROBLEMS TO BE SOLVED

[0007] The present invention relates to a remote eye gaze cursor control system embedded within a digital communication tool, such as a web-app, or a virtual conferencing platform, or a mobile app, or an application used for communication and interaction in digital formats. This system includes creating a virtual session or a broadcast session where the host or the operator and participant or user can join in to and which broadcasts remote estimation of the participants eye gaze. This application can be used for individuals who are neurodiverse for use during online telehealth sessions. This invention may comprise 3 main modules: (1) WebSocket or API connection which the web-app mounts to, (2) face and eye detection, and (3) cursor control estimation, that will be described herein. For simplicity, each of these modules have been summarised in the following three paragraphs. For the purposes of this report the terms clinician, host,operator and author are all used interchangeably, and the terms participant, student, and user, the same.

[0008] The present invention can either be packaged as a standalone web-app on a server accompanied by an audio-visual virtual conferencing platform, or be embedded as an application or tool which can be deployed inside a teleconferencing platform such as Coviu. The intention for both system deviations may be the same; so the clinician and user can (through virtual means) audio-visually communicate while also being able to communicate through the present invention as a live broadcasted view of the user’s eye gaze onto the main session window. At a high level, both systems, and therefore the present invention, operate using a workflow that can be summarised as follows:1 ) The clinician launches a session which generates a unique key and uploads or shares the required session content,2) The key can then be used by the user / s to join into the session,3) The user may go through a calibration session (the Calibration Window) prior to beginning the testing session to calibrate the eye tracking algorithm and validate eye tracking performance,4) Optionally, the user may go through a testing session (the Testing Window) prior to beginning the main session to assess the accuracy of the eye tracking performance,5) Once both the clinician and user are remotely dialled into the same main session (the Main Session Window), the eye tracking can commence along with all other features included in the main session,6) The user’s eye gaze controlled cursor is broadcast onto the clinician’s screen during the main session so that the clinician can see what the user is looking at on the screen at any instant.7) The clinician may end the main session for both the clinician and user at any time, remove or “kick” the user from the main session, or upload or share new main session content.The precise nature of these steps or their order may be different and unique acrossplatforms and sessions. The precise details of what the host and user concurrently see during the Calibration, Testing, and Main Session Window may be different and unique or the same. The nature of this is to ensure the host / operator is able to remotely assist the user, specifically one with a disability or a neurodisability to successfully use the technology. While it may be preferable for a user with a disability or a neurodisability from using this system, it may be appreciated that a user without any disability or a clinician can use the system and / or test out the system. In this same sense, the host / operator and the participant / user may be able to control features, settings, view, and options on the participant’s display from time to time and in the case the user is unable to complete these actions themselves. In this same sense, the host / operator and participant / user may also be given alerts and / or prompts in the form of audio, visual and / or text to alert and / or prompt the host / operator and participant / user of reduced eye gaze performance or calibration among other things necessary for the performance of the eye gaze session. Other games, accessories and settings may be accessed and used by the clinician and / or user during the main session. The main session may also have a classroom functionality where multiple users can join one or multiple clinician’s main session(s) and for example, the eye gaze controlled cursor would be labelled for each user. The calibration step may also be accessed at any time during the session by the clinician and / or user.

[0009] In the present invention, the remote eye gaze algorithm estimates the direction of the user’s eye gaze to control a cursor during a calibration session. A cursor can comprise either the participant’s own cursor, the host’s cursor, or an additional cursor displayed in the calibration session during eye tracking. This cursor can either be projected to both the host and participant’s Testing or Session Window or just the host’s / operator’s screen to make an assessment of where the participant / user is looking. For example, the user’s eye gaze controlled cursor may be broadcasted live to the clinician’s web page such that the clinician can observe where the user is looking on the screen at any instant in time. Initially, the system captures the video-feed through the user’s in-built PC camera, or external webcam, or the user’s in-built mobile camera. From the camera stream input, the present invention is able to capture a real-timeestimation of the user’s face and eyes. To ensure accurate face and eye detection, the user must have their face positioned from the webcam such that the user’s head and face is framed within the camera’s field-of-view and be positioned in good lighting conditions. The present invention may also comprise of other computer vision techniques to enhance facial and eye recognition such as pose estimation, calibration and / or other signature mapping techniques. During the calibration phase, the present invention may give audiovisual prompts to coach the user into the right position for accurate eye gaze tracking as well as verification of eye tracking performance before entering the main session. For an example of the former, a fill-the-oval thumbnail display may be used to prompt the user into the same position relative to the camera. For an example of the latter, an eye tracking check or test (Testing Window) and / or a percentage score of calibration accuracy, and a suggestion to repeat the calibration or continue onto the session. During the calibration session, the present invention may give the user and / or host / operator access to re-do or repeat the calibration at any point.

[0010] The present invention estimates eye gaze direction based on the images of the eyes. This is attained from the live video feed of the user’s integrated or external computer camera. The eye gaze controlled cursor is then mapped to the equivalent position the user is looking at on the computer’s display. This cursor may then be broadcast to the clinician and / or user for a remote assessment of where the user is looking at on the screen. The current invention has accuracy to control the cursor to a grid of 3x3 on the display but may achieve accuracy of a grid sized 8x8 or higher. The current invention may also include a dwell or blink function which would allow the user to interact with the Testing and / or Main Session Window. After a predefined period of dwell or number of sequential blinks, set as default or pre-set by the user or clinician, the eye gaze controlled cursor would create an execution on the object, feature, button or something similar that the user is gazing at. This may come packaged in the form of interactive games, tools or accessories available for the host / operator and / or user to choose from during the testing session.

[0011] The proposed invention will ensure accessibility and affordability as an assistive technology as it only requires a built-in computer webcam or existing hardware setups instead of costly physical eye tracking devices or technologies.

[0012] The proposed invention allows clinicians to remotely view, assess and communicate with user’s using eye gaze.

[0013] As a consequence of the above two mentioned advantages and as an advantage in and of itself, the proposed invention intends to enhance quality of life outcomes for people with a disability or a neurodisability by providing rehabilitation, communication, and education as well as health and social care services remotely through the use of eye tracking. Neurodisability may be a condition associated with impairment involving the nervous system and can include conditions such as cerebral palsy, autism or epilepsy.

[0014] It is an object of the present invention to overcome or ameliorate at least one of the disadvantages of the prior art, or to provide a useful alternative.

[0015] MEANS FOR SOLVING THE PROBLEM

[0016] A first aspect of the present invention may relate to a system adapted for interacting with a user from monitoring the activity of the user’s eye gaze, the system comprising: a user webcam configured to determine a location of a user’s eye gaze on a user display configured to display digital content to the user; an operator display configured to see the live interaction of the user’s eye gaze from movement of the eye gaze controlled cursor projected in the operator display; a processor with a gaze estimation module, the processor configured to: creating a calibration session for the user to join, wherein when joined, the gaze estimation module receives eye-motion signals from the user webcam as calibration data to use to estimate eye gaze; estimating user gaze direction and focus to predict where the user is looking at on the display, wherein the predicted user eye gaze is represented by movement of the eye gaze controlled cursor in the operator display and the user display; creating a testing session for the operator and the user to test eye gaze accuracy; and creating a main session for the operator and theuser to join; generating an interactive stimulus via user gaze-based interaction or blinkbased interaction, wherein a predefined period of gaze-based interaction or blink-based interaction allows for the eye gaze controlled cursor to execute on the interactive stimulus; analysing and determining user focus based on eye tracking from interaction of the interactive stimulus.

[0017] Preferably, the processor provides user eye tracking accuracy calibration data to the operator display and the user display.

[0018] Preferably, the system provides a step of calibrating user eye tracking until the calibration data passes above an accuracy threshold, wherein the accuracy threshold is above 50% for passing calibration. More preferably, the accuracy threshold is above 75% for passing calibration.

[0019] Preferably, the calibration session allows the operator and the user access to repeat the calibration.

[0020] Preferably, when passing calibration, the processor sends calibration data to the operator display whether to recalibrate or to initiate the testing session or the main session.

[0021] Preferably, the processor further comprising a cursor tracking engine, wherein the processor when receiving permission from the host to access the user cursor control access, the cursor tracking engine begins tracking cursor control, and wherein when the host permission is not received, the system continues to passively calculate user eye gaze.

[0022] Preferably, the user display has a first group of interface controls, and wherein the operator display has a second group of interface controls; wherein amount of the second group of interface controls is greater than the first group of interface controls, and wherein the second group of interface controls consists of the first group of interface controls; and wherein the second group of interface controls consists of the first group of interface controls such that the operator can control the user’s display.

[0023] Preferably, the user display has a first group of buttons comprising: a user video live stream display button, ending the virtual telehealth session button, turning on or off user video footage button, screen grabbing button, turning on or off microphone audio button.

[0024] Preferably, the operator display has a second group of buttons comprising: a user video live stream display button, ending the virtual telehealth session button, turning on or off user video footage button, screen grabbing button, turning on or off microphone audio button, a calibration button, a share screen button, a user track eyes button.

[0025] Preferably, when the operator selects a button, the corresponding user button feature starts, stops or updates.

[0026] Preferably, the calibration session is broadcast, and wherein the testing session is broadcast, and wherein the main session is broadcast.

[0027] Preferably, when a second user joins the calibration session or the testing session or the main session, wherein the predicted second user eye gaze is represented by movement of a second eye gaze controlled cursor in the operator display and the user display.

[0028] Preferably, the second eye gaze controlled cursor for the second user is different to the eye gaze controlled cursor for the first user.

[0029] Preferably, the user is one selected from the group of: a normal user, a clinician, a user with a disability, a user with neurodisability; and wherein the operator is one selected from the group of: a normal user, a clinician, a user with a disability, and a user with neurodisability.

[0030] Preferably, the digital content is at least one selected from the group of: interactive content, and static content.

[0031] Preferably, the system is embedded within a digital communication tool.

[0032] Preferably, the digital communication tool is one selected from the group of: a web-app, a virtual conferencing platform, a mobile app, an application used for communication and interaction in digital format.

[0033] Preferably, when the user focus falls below a predetermined attention threshold, the processor is configured to send an alert to the operator display and to the user display.

[0034] Preferably, the alert is at least one chosen from the group of: a visual alert, an audio alert, and a text alert.

[0035] Preferably, the alert stops once the predetermined attention threshold is met.

[0036] Preferably, the alert is not stopped over a predetermined period, an in-person helper alert is sent to a closest helper to the user.

[0037] Preferably, the host selects a button, the corresponding user button feature either start or stop.

[0038] Preferably, when the user turns on user video live stream, the system can allow the host to turn off the user video live stream.

[0039] Preferably, when the user turns off user video live stream, the system does not allow the host to turn on user video live stream.

[0040] Preferably, the calibration session is virtual.

[0041] Preferably, the testing session is virtual.

[0042] Preferably, the user gaze-based interaction or blink-based interaction is at least one selected from the group of: a predefined period of dwell, and a predetermined number of sequential blinks

[0043] Preferably, the operator and the user can each control features on the user display during any session.

[0044] Preferably, the control features on the user display is at least one selected from the group of: audio and visual controls, webcam zoom control, webcam tilt control, webcam pan camera control, toggling on and off eye gaze monitoring, launching calibration session, and controlling user session view.

[0045] Preferably, the user can use eye gaze monitoring to interact with features on the operator’s broadcasted display, wherein the feature is one selected from the group of: eye gaze apps, eye gaze games, session content, and gaining remote access and control of the operator or the user’s display.

[0046] Preferably, the system allows for the eye gaze monitoring to switch between the operator and the user.

[0047] Preferably, aspect ratio of the user’s display is broadcasted to the operator and illustrated through a graphic element.

[0048] In the context of the present invention, the words “comprise”, “comprising” and the like are to be construed in their inclusive, as opposed to their exclusive, sense, that is in the sense of “including, but not limited to”.

[0049] The invention is to be interpreted with reference to the at least one of the technical problems described or affiliated with the background art. The present aims to solve or ameliorate at least one of the technical problems and this may result in one or more advantageous effects as defined by this specification and described in detail with reference to the preferred embodiments of the present invention.BRIEF DESCRIPTION OF THE FIGURES

[0050] FIG. 1 is a block diagram of an example software architecture for facilitating eye tracking control, according to some embodiments.

[0051] FIG. 2 is a state diagram of example states to describe the behaviour of the system capable of facilitating eye tracking control, according to some embodiments.

[0052] FIG. 3 is an interface diagram depicting an example user interface displaying objects within quadrants to accompany eye tracking control and remote assessment of eye gaze, according to some embodiments.

[0053] FIG. 4 is a block diagram of an example flow of data used to facilitate eye tracking control according to some embodiments.

[0054] FIG. 5 is a flowchart of an example method of a session and facilitating eye tracking control, according to some embodiments.

[0055] FIG. 6 is an interface diagram depicting an example user interface displaying objects to make clear the usage of function within a telehealth session to facilitate eye tracking control and remote assessment of eye gaze, according to some embodiments.DESCRIPTION OF THE INVENTION

[0056] Preferred embodiments of the invention will now be described with reference to the accompanying drawings and non-limiting examples.

[0057] Example systems and methods to facilitate eye tracking control are described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of example embodiments. It will be apparent to those skilled in the art that modifications, including additions and / or substitutions may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the disclosure is written to enable one skilled in the art to practice the teachings herein without undue experimentation.

[0058] In accordance with various embodiments of the present invention, the method and system for delivering and managing remote eye gaze cursor control technology uses a combination of face and eye detection, cursor control, and WebSocket connection which a web-app mounts to. The invention may be run as an application integrated on a telehealth platform or as a standalone application with a built-in virtual conferencing component. For example, the former, as a plugin feature on Coviu which can be run and used during telehealth sessions, or the latter, as a web application where an audio-visual clinician-patient session can be created, hosted and accessed via the browser. It may be appreciated that a plugin feature can be integrated into for example, FaceTime™ or another virtual communication tool can advantageously enhance functionality and provide additional capabilities. The integration into these communication tools may be done through API integration, software development kits, browser extensions when the tool is accessed through a browser, and / or custom user interface enhancements by adding buttons, menus or toolbars that provide access to the plugin’s features. The intention for both system deviations may be the same; so the clinician or operator and user can (through virtual means) audio-visually communicate while also being able to communicate through the present invention as a live broadcasted view of the user’s eye gaze onto the session window. The eye gaze may be the direction and / or the period of the user’s gaze. The invention may require a minimum of two persons: 1) the host or operator (the one hosting or operating the session), and 2) the user (the one participating in the session), similar to how a typical virtual conferencing platform functions. The invention may also allow a classroom style session with one or multiple hosts and multiple users. In this sense, each user may have their eye gaze controlled cursor labelled during the session to discriminate what each user is looking at. The roles of the host and user(s) during the session are as follows:• The host or clinician or operator may: o create a unique session, o upload the session content onto their shared display, o start the eye tracking algorithm, start the user’s calibration, and set up all user and session settings and controls o have their mouse cursor cast onto the user’s display,o and control / facilitate the session.• The user may: o direct and control movement of a cursor (the host’s / operator’s, user’s or a new cursor) depending on where the user is looking on the display which is broadcast to the host, o interact with the shared display from the host or operator via eye gaze controlled cursor movement and ‘clickable’ interactions.

[0059] In this sense, and for the purposes of the invention, the invention may be largely controlled and facilitated to the extent of all or some embodiments, by the clinician or operator such that operation of the invention may be a ‘hands-free’ experience for the user. As a consequence and as a separate point altogether, if it wasn’t already apparent in previous descriptions, the host or the operator and user may see the same display screen at any one time, as you would with a ‘share screen’ function during a virtual conferencing session.

[0060] In accordance with various embodiments, a user of a session may interact with and control a cursor on the display through the user’s eye movement, such that both the host or the operator and / or user may see the eye gaze controlled cursor. An image of the user’s eyes and face, captured by a front-facing camera, on or coupled to the user’s computing device, may be analysed using our system’s computer vision eye gaze detection algorithm. For example, the captured images may be processed to extract information relating to features of the user’s eyes and / or face. The system may include other features as it relates to eye and face detection during image capture and analysis, such as pose estimation and calibration or other signature mapping techniques in order to better discriminate ocular characteristics of the user. The invention may then use this information to estimate the location on the display at which the user is looking at any instance in time. For example, within a session, the invention may be able to estimate at which displayed icon or content material the user is looking. The estimation of where the user is looking may be used to direct a cursor. The eye gaze controlled cursor is broadcast live to the host / operator such that they can remotely see where the user is looking at any time. For example, the user may direct and control movement of a cursoron the display, once access has been granted, depending on the region of the display the user is looking at and the host / operator can observe this remotely on their display.

[0061] In some embodiments, the host or the operator may share session contents, including sharing their screen or uploading specific session material. For example, for the latter, the host or the operator may upload a pdf document which includes all the contents used for that session. If this is the case, the host or operator may control / navigate the pdf document and the host or the operator and the user may see the same section of the pdf document at the same instant. The host or the operator may also have the option of uploading new session material during the session. Additionally, the host or the operator may choose to use the interactive eye gaze games, tools and accessories available for the session for the host / operator and / or user to use. These may be designed with the intention of communication, education and / or research purposes. These may also be customised by the host / operator and / or user to suit the requirements and context of the session.

[0062] In some embodiments, the user may interact with and direct a cursor to the objects being displayed on the Session Window using the eye tracking feature. The objects of interest (e.g., the display objects being gazed upon by the user) may be presented to the user through screen sharing by the host or other functionality which displays the session content for both host / operator and user. The user may interact with these objects by hovering the cursor over the objects. In some embodiments, a dwell or blink function may be included when eye gaze cursor control is active which will allow the user to interact with and select objects of interest displayed on the Session Window. After a predefined period of dwell or sequential blink the cursor would click on the object the user is looking at. These predefined settings may be adjusted by the user and / or host / operator or left as default. These predefined settings may be adjusted by the user and / or host / operator or left as default. These settings may be part of a large accessible menu of settings changes to customise the session and calibration. For example, this may include customising the interactive display icons, and colour changes to those with sight impairment.

[0063] In some embodiments, the session may begin with a calibration (the Calibration Window) of the user’s eyes. For example, a 9-point calibration where the user may look at 9 dots around the region of the screen. The calibration window or the calibration session may give the user prompts and coaching on how to improve the accuracy of the eye tracking estimation. The calibration window may also complete a validation check of the user’s eye tracking accuracy and performance before entering into the main session or the testing session. It may then give the user the option to repeat calibration or enter into the main session. The calibration may be initiated, set-up with preferred settings, and repeated by both the user and host / operator.

[0064] In some embodiments, the activation signal may start an eye control mode on the testing Session Window (e.g., when the session is created and the user joins), which may last until the activation signal terminates (e.g., when the host / operator turns off or deselects the application button or ends / leaves the session). During the time the activation signal is active, the invention may run our system’s eye gaze detection algorithm. The cursor control signal may be accessed while the activation signal of the eye control mode is turned on to give the user access to control a cursor depending on eye gaze. For example, during the session and while the eye control mode is active, the host / operator may turn on the cursor control button to begin cursor control through eye gaze. The cursor control toggle or button may appear in any manner, such as on a specific area of the display, using a button or other method to access and run these features, and the labelled name of these buttons.

[0065] In some embodiments, the invention may use a WebSocket connection, which makes possible an open two-way interactive communication session between the user and host’s / operator’s browser and the server. In these embodiments, the eye tracking may be performed locally on the client side and relevant information including eye gaze position relative to the aspect ratio of the display may be set within a database on the server side. For example, as a standalone web-app firebase realtime database may be used for the ability to monitor changes to the database in real time including granting host authorship, viewing session content from both the host and user’s browser, and broadcasting user eyegaze controlled cursor position and movement to the clinician or the operator. These functionalities may be equivalent as an app inside a teleconferencing platform, but may be executed via an API or other means.

[0066] FIG. 1 is a block diagram of an example software architecture 100 for facilitating eye tracking control. The software architecture 100 may be divided into different software layers. The bottom layer may include a webcam module 114 and a computer monitor 116 that may correspond to the respective hardware (e.g., a built-in / integrated webcam or external webcam and monitor). A camera layer may include a camera control module 110 that may be in charge of communicating with the webcam in order to perform camera operations such as, for example, starting the camera, grabbing images, controlling the camera properties, and the like. This layer may also include a camera and light sync module 112 that may synchronise to adjust for ambient light conditions in order to improve tracking of the user’s eyes. The eye tracking layer contains an eye detection and tracking module 108 which may process the video stream to find features of the face and eyes as well as better discriminate between user facial characteristics such as pose estimation and calibration. These features may be used by the gaze estimation module 106 in the gaze estimation stage, which may be in charge of making predictions of where the user is looking at on the display. The API layer may be used for communication between the eye tracking layer and the application layer. The API module 104 may send live eye gaze prediction data from the eye tracking layer used during sessions to the application layer as well as other information related to the user’s eye gaze and requirements of a virtual session. The application layer contains the application module 102 which may use eye gaze information to broadcast information to the user and host / operator about the location the user is looking at any time instant. For example, this information may include control of a cursor (the host’s / operator’s, user’s or another cursor). At the same time, the application layer may deploy commands (e.g., start and stop cursor control). Additionally, other functionalities within the application may include an audio-visual platform with virtual conferencing capabilities, screen sharing and interactive eye gaze features and games, remote cursor control access, and other settings changes. The application layer may be the layer at which the host / operatorand user interfaces with in order for the user to use the eye gaze cursor control technology. There may be intermediary steps within each layer to assist in communicating with the next but the system defined in FIG. 1 contains the high order modules and order of operations that may be carried out within the invention to achieve the desired effect.

[0067] FIG. 2 is a state diagram of example states 200 to describe the behaviour of the system capable of facilitating eye tracking control. The diagram 200 may be represented but not limited to six states. These states may be used as a key to represent the range of movement of the cursor control within the field of view. The user eye gaze input may include a centre portion 202 and corresponding cursor position on the monitor. The user eye gaze input may also include a right 204, left 206, bottom 208 and top 210 portion and corresponding cursor positioning. The user eye gaze input may also include a diagonal portion along multiple axes with one possible diagonal position 212 and corresponding cursor position shown.

[0068] FIG. 3 is an interface diagram depicting an example user interface displaying objects to accompany eye tracking control and remote assessment of eye gaze. While a specific example 300 is shown, the technology is not limited by this, and any configuration or combination of configurations may be implemented. In some embodiments, settings associated with increased spatial divisions and object display, and the like, may be configured by the host or user. The example display 300 shows a presentation given by the host or the operator via share screen or other screen mirroring functionality to the user. The presentation is divided into four quadrants 302 with a visual cue in each. The user is given remote access to control a cursor 304 with eye gaze. The division the cursor rests in is an indication of where the user is looking. If an interactive element is available for the user to select and after a predefined period of time that the cursor hovers over an object of interest, named dwell function, the mouse cursor may click on the selected object, in this care the upper left tile.

[0069] FIG. 4 is a block diagram of an example flow of data 400 used to facilitate eye tracking control. A session may be established between the host / operator and the userand the user’s video stream data 402 during the session may be obtained. The video stream data may be fed into the eye tracking engine 404 and the face and eyes may be detected using the face and eye detection model and predictions are made about what the user is looking at on the display. This face and eye prediction information may be transmitted via the API 406 or may be by other means to the session window. The host or the operator may give remote access to the user for cursor control through eye tracking technology. The cursor tracking may move according to the user’s eye gaze. In this depiction 408, the user is looking to the right of the display and the cursor is broadcast live to the user and host’s session window on the right side of the display.

[0070] FIG. 5 is a flowchart of an example method of a session and facilitating eye tracking control. The flowchart depicts a method 500 or system 500 in which the user’s eye gaze may be used in combination with a host / operator input to control and direct a cursor, associated with the user’s eye gaze. In this method 500 or system 500, the eye tracking engine may be tracking the user’s gaze when activation occurs through initiating a session. In operation 502, the application is started by the host / operator and a unique session token is generated. In operation 504, the user joins remotely using the unique session token and the live video stream data is received from the user’s integrated computer camera. In operation 506, when the host / operator input is received, the user and host / operator may enter the calibration window or the calibration session which calibrates the user’s eye gaze. In operation and after calibration 508, information about the eye tracking accuracy is presented, which may be part of the calibration data; and the user and host / operator are given the opportunity to repeat the calibration or enter into the session window or the testing session. In operation 510, the host / operator shares the testing session contents with the user. In operation 512, the user’s eye gaze is calculated. In operation 514, when the host / operator input is received, the user eye gaze controlled cursor is accessed. In operation 516, once the host / operator has given command input for user cursor control access, the cursor tracking engine may begin tracking cursor control as it relates to eye gaze and is broadcast to the host / operator and user. If the host / operator input command is not given, the application may continue to passively calculate user eye gaze. In operation 518, when the host / operator input is received, the user eye gazecontrolled cursor stops and the cursor is no longer broadcast to the host / operator and user. In operation 520, when the host / operator input is received, the application can be ended for either the host / operator and user or just the user and the eye tracking engine will stop. Although it’s not represented on the flowchart, calibration can be repeated and the application can be ended by the host / operator at any point during the session.

[0071] FIG. 6 is an interface diagram depicting an example user interface 600 displaying objects to make clear the usages of function to facilitate eye tracking control and remote assessment of eye gaze during a telehealth session or a testing session. While a specific example is shown in FIG 6, the technology is not limited by this example, and any configuration or combination of configurations may be implemented. Features and button displayed as shown in FIG. 6 should only be treated as a representation of some basic instalments that may be used to facilitate eye tracking control and remote assessment of eye gaze and not a limitation of this technology. Much of this display is dependent on the telehealth platform the system is interfacing with and should not be a consideration nor have any bearing on the nature and scope of the current invention. This may include a video live stream display 602, buttons available to hang up the telehealth session 604 or the testing session 604, turn off / on video footage 606, screen grabbing 608, and turn off / on microphone audio 610. FIG. 6 is represented from the point of view of the host or the operator. This may be different from the point of view of the user by the available tools for selection 612. For the host or the operator, this may include a calibration button 614, a share screen button 616, and a track eye button 618. Once the host or the operator selects a button the corresponding button feature may start or stop. For advantageously improving and respecting the privacy of the user over conventional systems, when the user turns off any of the buttons, the system does not allow the host or the operator to turn on or to start the corresponding button feature. While the system provides a user improvement, in blocking a host or the operator to not be able to view the presence of the user in either the calibration session and / or the testing session, wherein repeated non-compliance associated with poor eye gaze accuracy results or data, may indicate that the user may not be doing the test or may not present during the session. In such cases, while the system provides a virtual session, for compliance reasons and / orwell-being or emergency reasons, the system may communicate to a qualified or registered helper, which may be the user’s emergency contact, to go for an in-person check up on the user. When the in-person check has been completed, and that the user is well to resume the session, the registered helper can then notify the operator that the session can be re-initiated with the user.

[0072] Although an embodiment has been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the technology. Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. The accompanying drawings that form a part hereof, show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This Description of Embodiments, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

[0073] Such embodiments of the invention subject matter may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

[0074] Although the invention has been described with reference to specific examples, it will be appreciated by those skilled in the art that the invention may be embodied in many other forms, in keeping with the broad principles and the spirit of the invention described herein.

[0075] The present invention and the described preferred embodiments specifically include at least one feature that is industrial applicable.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. A system adapted for interacting with a user from monitoring the activity of the user’s eye gaze, the system comprising: a user webcam configured to determine a location of a user’s eye gaze on a user display configured to display digital content to the user; an operator display configured to see the live interaction of the user’s eye gaze from movement of the eye gaze controlled cursor projected in the operator display; a processor with a gaze estimation module, the processor configured to: creating a calibration session for the user to join, wherein when joined, the gaze estimation module receives eye-motion and head-motion signals from the user webcam as calibration data to use to estimate eye gaze; estimating user gaze direction and focus to predict where the user is looking at on the display, wherein the predicted user eye gaze is represented by movement of the eye gaze controlled cursor in the operator display and the user display; creating a testing session for the operator and the user to test eye gaze accuracy; and creating a main session for the operator and the user to join; generating an interactive stimulus to the user display; interacting with the interactive stimulus via user gaze-based interaction or blink-based interaction, wherein a predefined period of gaze-based interaction or blink-based interaction allows for the eye gaze controlled cursor to execute on the interactive stimulus;analysing and determining user focus based on eye tracking from interaction of the interactive stimulus.

2. The system of claim 1, wherein the processor provides user eye tracking accuracy calibration data to the operator display and the user display.

3. The system of any one of claims 1 to 2, calibrating user eye tracking until the calibration data passes above an accuracy threshold, wherein the accuracy threshold is above 50% for passing calibration.

4. The system of claim 3, wherein the accuracy threshold is above 75% for passing calibration.

5. The system of any one of claims 3 to 4, wherein the calibration session allows the operator and the user access to repeat the calibration.

6. The system of any one of claims 3 to 4, wherein when passing calibration, the processor sends calibration data to the operator display whether to recalibrate or to initiate the testing session or the main session.

7. The system of any one of claims 1 to 6, wherein the processor further comprising a cursor tracking engine, wherein the processor when receiving permission from the host to access the user cursor control access, the cursor tracking engine beginstracking cursor control, and wherein when the host permission is not received, the system continues to passively calculate user eye gaze.

8. The system of any one of claim 1 to 7, wherein the user display has a first group of interface controls, and wherein the operator display has a second group of interface controls; wherein amount of the second group of interface controls is greater than the first group of interface controls, and wherein the second group of interface controls consists of the first group of interface controls; and wherein the second group of interface controls consists of the first group of interface controls such that the operator can control the user’s display.

9. The system of claim 8, wherein the user display has a first group of buttons comprising: a user video live stream display button, ending the virtual telehealth session button, turning on or off user video footage button, screen grabbing button, turning on or off microphone audio button.

10. The system of any one of claims 8 to 9, wherein the operator display has a second group of buttons comprising: a user video live stream display button, ending the virtual telehealth session button, turning on or off user video footage button, screen grabbing button, turning on or off microphone audio button, a calibration button, a share screen button, a user track eyes button.

11. The system of claim 10, wherein the operator selects a button, the corresponding user button feature starts, stops or updates.

12. The system of any one of claims 1 to 11, wherein the calibration session is broadcast, and wherein the testing session is broadcast, and wherein the main session is broadcast.

13. The system of any one of claims 1 to 12, wherein a second user joins the calibration session or the testing session or the main session, wherein the predicted second user eye gaze is represented by movement of a second eye gaze controlled cursor in the operator display and the user display.

14. The system of claim 13, wherein the second eye gaze controlled cursor for the second user is different to the eye gaze controlled cursor for the first user.

15. The system of any one of claims 1 to 14, wherein the user is one selected from the group of: a normal user, a clinician, a user with a disability, a user with neurodisability; and wherein the operator is one selected from the group of: a normal user, a clinician, a user with a disability, and a user with neurodisability.

16. The system of any one of claims 1 to 15, wherein the digital content is at least one selected from the group of: interactive content, and static content.

17. The system of any one of claims 1 to 16, wherein the system is embedded within a digital communication tool.

18. The system of claim 17, wherein the digital communication tool is one selected from the group of: a web-app, a virtual conferencing platform, a mobile app, an application used for communication and interaction in digital format.

19. The system of any one of claims 1 to 18, wherein when the user focus falls below a predetermined attention threshold, the processor is configured to send an alert to the operator display and to the user display.

20. The system of claim 19, wherein the alert is at least one chosen from the group of: a visual alert, an audio alert, and a text alert.

21. The system of claim 20, wherein the alert stops once the predetermined attention threshold is met.

22. The system of claim 21, wherein the alert is not stopped over a predetermined period, an in-person helper alert is sent to a closest helper to the user.

23. The system of any one of claims 1 to 22, wherein the user gaze -based interaction or blink-based interaction is at least one selected from the group of: a predefined period of dwell, and a predetermined number of sequential blinks.

24. The system of any one of claims 1 to 23, wherein the operator and the user can each control features on the user display during any session.

25. The system of claim 24, wherein the control features on the user display is at least one selected from the group of: audio and visual controls, webcam zoom control, webcam tilt control, webcam pan camera control, toggling on and off eye gaze monitoring, launching calibration session, and controlling user session view.

26. The system of any one of claims 1 to 25, wherein the user can use eye gaze monitoring to interact with features on the operator’s broadcasted display, wherein the feature is one selected from the group of: eye gaze apps, eye gaze games, session content, and gaining remote access and control of the operator or the user’s display.

27. The system of any one of claims 1 to 26, wherein the system allows for the eye gaze monitoring to switch between the operator and the user.

28. The system of any one of claims 1 to 27, wherein aspect ratio of the user’s display is broadcasted to the operator and illustrated through a graphic element.

Images