Communication method and system

Abstract

To provide a control system that interfaces with an individual through tracking eyes and / or tracking other physiological signals generated by the individual.SOLUTION: A system 100 classifies captured eye images into gestures. The gestures emulate joystick-like control of a compute. These gestures permit a user to operate, for instance a computer or a system with menu items.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The present disclosure relates to a system that allows an individual to operate a communication module or other utility by tracking any one or combination of the individual's eye or other physiological parameters. [Background technology]

[0002] Systems and methods are known that allow communication with a user by tracking the user's eyes.

[0003] WO2016142933 discloses such a system with a selection interface that selectively presents a set of communication options to a user. An optical sensor detects light reflected from the user's eye to provide a correlation signal that is processed to determine an orientation of the eye relative to the user's head. Based on the determined relative eye orientation, a selected communication option is determined and implemented. Summary of the Invention

[0004] The present disclosure relates to a computerized system that interfaces with an individual by tracking the eyes and / or by tracking other physiological signals generated by the individual. In other words, it is a system that includes a utility having a control unit associated with a camera configured to track eye and / or eyelid movements, and may further include a utility configured to track another physiological parameter. The system is configured, according to an embodiment, to classify captured eye images into gestures, where the gestures are pupil position or gaze direction, a series of directional eye movements, blinking of the eyelids, etc. These gestures allow the user to operate, for example, a computer or system with menu items. In this way, the eye movements can be, for example, a user to navigate a menu, move a cursor on a screen with eye movements, fixate the eyes in a particular position at a given time, or the system can also be configured to classify other physiological data and convert them into computer-readable commands or to operate one or more peripheral devices. For example, the system may perform defined eye gestures, perform predefined breathing patterns, move a cursor on a screen by moving a body part, select an item by a breathing motion, or perform electrophysiological stimulation. In general, the system of the present disclosure allows a user to operate a computer in a manner similar to that of a joystick. With the joystick-like approach of the present disclosure, the only reference point is the camera capturing an image of the eye, and there is no need to detect the exact position or location where the user is looking or at the corneal reflex. Also, according to the present disclosure, there is typically no need for a calibration procedure using the screen before use (indeed, there is no need to use the screen to communicate using the system).

[0005] According to some embodiments, menus are not presented to the user on a screen, and the user's navigation through menus and item selection is performed without a screen (e.g., based on predetermined or previously introduced or presented menus to the user).

[0006] According to some embodiments, the menu is presented to the user on the screen only during an initial introductory phase during which feedback or instructions are provided to the user, for example, any time frame on the scale of minutes, hours, weeks and months.

[0007] According to some embodiments, gestures that allow a user to operate a computer are general viewing in a defined direction (eye gestures) rather than gestures in which the user is looking at a specific location (gaze). For example, a general left gaze works as a gesture even if the user is not directing their gaze at a specific physical or virtual object.

[0008] Generally, the operation of the disclosed system is independent of lighting conditions.

[0009] According to some embodiments, the classification of gestures is based on the use of machine learning techniques. In particular, the machine learning model is a neural network model composed of multiple linear transformation layers followed by element-wise nonlinearities. The classification can include eye characterization of individual users or across users. According to some embodiments, the classification estimates the range of eye movements.

[0010] According to one embodiment, the system allows an individual using the system (herein "user") to navigate through menus presented to the user. The menus may be presented audibly (through speakers, earphones, headphones, embedded audio devices, etc.) or visually (through an on-screen display, a small display in front of the individual, etc.). The menus are hierarchical, i.e., selection of a menu item opens other lower-level selectable options. As an example, a higher level of selectable menu items may allow the user to select one of several groups of letters (e.g., one group consisting of letters A-F, a second group of letters G-M, etc.), and once selected, the user is presented with the ability to select one of another group of letters (e.g., when the first group is selected, the user is given the option to select from A, B, C, or D-F, and when D-F is selected, the user is presented with the individual letters for their selection). However, selection may also be prompt-driven, i.e., the user is prompted to move their eyes in a particular direction for a particular selection.

[0011] According to some embodiments, the menu items, or portions thereof, can be customized to suit particular needs, which can be accomplished locally, for example via a user or caregiver interface, or remotely, for example by a remote server.

[0012] The menu items or parts thereof may be suggested to the user by the system or control unit. The menu items or parts thereof may also be suggested or presented to the user based on input received from the surrounding environment.

[0013] In some embodiments, the control unit receives and processes speech data (e.g., by natural language processing). For example, when the user asks a question of another person, such as a caregiver, the control unit receives and processes the doctor's speech and suggests a response based on a contextual analysis of the other person's speech. The disclosed system is useful for allowing paralyzed individuals who are otherwise unable to communicate to communicate with their surroundings, including caregivers and peripherals such as alarm systems, audiovisual systems, and computers. One targeted group of individuals are ALS patients, who, as their disease progresses, lose the ability to move their limbs and other muscles, as well as the ability to speak and display sounds. The system is also useful for individuals with temporary communication impairments, such as intensive care unit patients, patients receiving temporary or permanent respiratory support, etc.

[0014] According to an embodiment of the present disclosure, there is provided a control system for operating a computer, comprising a camera configured to continuously capture images of one or both of a user's eyes and eyelids and generate image data representative thereof, and a control unit in data communication with the camera and the computer, the control unit operable to receive and process the image data and classify them into gestures intended to mimic joystick-like controls of the computer.

[0015] The term joystick-like control as used herein refers to a classification of gestures that involve tracking the position of the pupil region.

[0016] A pupil area in the context of this disclosure is the pupil or any portion thereof that is identified as indicative of the pupil.

[0017] In some embodiments, the location of the pupil region is determined based on a database including image data with labeled gestures, said image data being obtained from the user himself or another user or group of users. In some embodiments, the location of the pupil region based on the labeled database is determined by utilizing machine learning techniques, for example models that consider the likelihood of certain image data corresponding to a particular gesture.

[0018] According to some embodiments, the location of the pupil region is determined based on its position in the threshold map, the particular location being determined whenever the pupil region touches the boundary or tangent to the boundary of the threshold map. For example, when the pupil region touches the top boundary of the threshold map, the image data is classified as an "up" gesture, or when the pupil region does not touch the boundary of the threshold map, the image data is classified as a "straight" gesture. The threshold map may be derived from a position map that includes an area that is within the range of motion of the pupil region. As an example, the position map is defined as a rectangle defined by the positions of the top, bottom, left edge, and right edge of the pupil region. In some embodiments, the threshold map covers an area bounded by boundaries that are at least 20%, 40%, 60%, 80%, 90%, 95% away from the center of the position map. The threshold map is typically at least 80% away from the center of the position map. The position map may be obtained based on the user's image data or a database that includes image data with or without labeled gestures. Optionally, the position map is within a larger region of interest (ROI) defined based on anatomical features of the eye or its surroundings.

[0019] According to some embodiments, the system requires the user to perform straight gestures in between other gestures.

[0020] According to some embodiments, a blink gesture is identified as a region of dark pixels.

[0021] According to some embodiments, a gesture is classified when the pupil region contacts a boundary or a tangent to a boundary of the threshold map for at least 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1, 2, 4, 8, 10 seconds.

[0022] In some embodiments, the camera is an infrared camera.

[0023] Furthermore, the control unit may be linked to one or more sensors for detecting other physiological measurements and is operable to receive and process physiological signals acquired by such devices or sensors and classify these into computerized commands.

[0024] For example, through eye movements (tracked by the position of the pupil relative to the eye socket) and optionally blinking of the eyelids, the user can navigate through and freely select selectable options. This includes navigating and selecting menu items, navigating and selecting hyperlinks, etc. According to one specific, non-limiting embodiment, directional movements or positions of the pupils can orient a cursor in a predefined direction, e.g., an upward position of the pupils can move the on-screen cursor upwards, a right position can move rightwards, etc., or alternatively, the user can define the direction of such cursor movement in addition to the system-defined directions. As a non-limiting embodiment, a blink of the user (or other gesture or physiological parameter defined by the user) initiates a camera operation, and a subsequent blink allows the user to browse through the selectable options and select an option by such blink. According to another example, menu items are voiced to the user, and when the desired menu item is voiced, the user blinks to select the desired item. According to another non-limiting embodiment, a user with limited abilities may operate the system using a single gesture based on their own definition, e.g., only a "left" gesture. Also, as an exemplary and non-limiting embodiment, the user is prompted by audio or visual output to select from several options, e.g., "up" (i.e., upwards) for one selection and "down" for another. According to a further exemplary and non-limiting embodiment, the user is presented with an option (e.g., through an audio readout) that prompts the user to gaze in a specific or non-specific direction for a predetermined time, perform a series of blinks, and close the eyelids when a particular choice is presented. The latter is useful, for example, for quickly selecting letters for writing.

[0025] According to some embodiments, the type of gesture, the number of gestures, the gesture duration, and any corresponding commands are defined by the user or caregiver.

[0026] In some embodiments, a series of 1, 2, 3 or 5 blinks may select the "call for help" option.

[0027] In some embodiments, a series of up to 10 blinks within a maximum of 30 seconds selects an item.

[0028] In some embodiments, closing the eyes for 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 30 seconds will cause the system to go into hibernation mode.

[0029] The control unit, according to an embodiment of the present disclosure, is configured to (i) receive and process the image data to identify and classify at least one of pupil position and eyelid movement into gestures, e.g., including one or more of pupil position, sequence of pupil position, and sequence of eyelid blinking, and generate gesture data, and (ii) utilize the gesture data to operate a computer. According to one embodiment, a computer linked to the control system operates a visual or audio output module. The output module enables a user to communicate with other individuals. In some embodiments, the audio output module is a bone conduction hearing aid.

[0030] Further, the control unit is configured to (i) receive and process physiological data and categorize them into commands, including, for example, any electrophysiological markers (e.g., recorded via an electroencephalography (EEG) device), somatosensory, respiratory, vocal, movement gestures, or any combination thereof, and (ii) utilize the physiological commands to operate a computer. According to one embodiment, a computer linked to the control system operates a visual or audio output module. The output module allows the user to communicate with other individuals.

[0031] For example, EEG signals can be recorded such that an EEG command initiates navigation in a time-dependent menu, and when the navigation menu reaches a desired menu item, the user can generate an additional EEG command that selects the menu item. According to another example, an EEG command triggers the initiation of the system.

[0032] Another embodiment of the present disclosure provides at least one sensor for measuring physiological parameters and a control unit configured for data communication with a computer, the control unit operable to receive and process physiological data acquired by the at least one sensor, classify and convert the same into commands, and send corresponding commands to the computer, thereby controlling the operation of the computer.

[0033] Another embodiment of the present disclosure provides an eye tracking based system including a camera, a first output module, and a control unit, typically including a computer or processor. The camera is operable to continuously capture images of one or both of a user's eyes and eyelids and generate image data representative thereof. The control unit is in data communication with the camera and the first output module and configured to (i) receive and process said image data to identify and classify at least one of pupil position and eyelid movement into gestures including one or more of pupil position, sequence of pupil position, and sequence of eyelid blinking, and generate gesture data, (ii) operate hierarchical user selectable menu items that allow a user to navigate and select menu items by said gesture data, and (iii) drive the first output module to present the menu items to the user. Optionally, the control unit is also configured in data communication with a sensor for measuring physiological parameters. The control unit is further configured to (i) receive and process physiological data from the sensors and categorize them into commands, (ii) operate a hierarchical user-selectable menu items to allow a user to navigate and select menu items via the commands, and (iii) drive a first output module to display the menu items to the user. The first output module is configured to provide one or both of a visual or audio presentation of the menu items to the user. In some embodiments, the audio presentation module is a bone conduction hearing aid.

[0034] Another embodiment provides an eye tracking based system including a camera, a first output module, and a control unit similar to that of the embodiment described in the paragraph above. The camera is operable to continuously capture images of one or both of a user's eyes and eyelids and generate image data representative thereof. The control unit is in data communication with the camera and the first output module. It includes a data processor configured and operable to process the image data in response to image data received from the camera, identify at least one of pupil positions and eyelid movements, classify them into gestures including one or more of pupil positions, sequences of pupil positions, and sequences of eyelid blinks, and generate gesture data. It also includes a menu generator module configured and operable to utilize the gesture data to manipulate hierarchical, user selectable menu items. The system further includes a first actuator module configured to drive the first output module to present menu items to a user via one or both of visual and audio presentations of the menu items, thereby enabling the user to navigate and select the menu items.

[0035] Optionally, the control unit comprises a data processor in data communication with a sensor for measuring a physiological parameter, configured and operable to respond to physiological data received from the sensor and to process and classify said physiological data into commands.

[0036] The gestures include straight, center, right, left, up and down positions of the pupil and blinks. Optionally, the gestures include a sequence of two or more eyelid blinks. For example, the correct position of the pupil can be classified as an "enter" command.

[0037] The gestures can be selected from any or a combination of eye gestures known in the art, for example, the gestures are fixations (static fixations) or a series of fixations and their durations, fixation points and clusters and their distribution.

[0038] The system (of any of the above embodiments) is operable to drive an output module arranged to output an alarm signal (typically an audio and / or visual signal).

[0039] The camera may be mounted in a holder that can be attached to the user's head, however, the camera may also be mounted on a frame close to the user, such as a bed frame, a frame carrying medical equipment, etc.

[0040] The menu items are arranged in a hierarchical manner. They may be arranged in successive layers of the hierarchy, such that, for example, selection of a first menu item allows the user to subsequently select a second menu item in a layer subordinate to the first menu item. Each such "layer" typically contains up to five selectable items, selectable according to the center, right, left, top, and bottom position of the pupil.

[0041] Additionally or alternatively, menu items may be selected via a prompt-based system, e.g., the user is instructed via visual or audio prompts to look in a particular direction to select one menu item, in a different direction for a second item, and so on.

[0042] In some embodiments, the menu items are user definable.

[0043] The system includes a driver for a second output module configured to generate an alert or to operate a peripheral system, such as, for example, a virtual assistant, a smart home device, a home air conditioning system, a television, a music player, a communication device, a wheelchair, a tablet, a smartphone, a gaming accessory, etc. The system is configured to operate such a second output module through a system-defined or user-defined specific gesture, such as a defined blink sequence.

[0044] The physiological measurements or data refer to any signals that may be obtained from the user's body, including any signals obtained from the user's neural, somatosensory, vocal, and respiratory systems, as well as selected muscle movements.

[0045] Such sensors for measuring physiological parameters are any sensor utility or measuring device, microphone, spirometer, galvanic skin response (GSR) device, touch or pressure probe, galvanic skin response probe (skin conductance probe), electroencephalogram (EEG) device, electroencephalography (ECoG) device, electromyography (EMG), electrocardiography (EOG), and devices for electrocardiogram. The data recorded by the sensors are classified into commands.

[0046] The command may be any one or combination of the following: a movement of any body part (e.g., a tap of a finger or pressing a response button), a breathing pattern, a smell, an audio output, a change in muscle tone, skin conductance, or a neural output.

[0047] The neural output may be, for example, a measured evoked response potential, or any marker related to the time or frequency of the measured data.

[0048] Users of the disclosed system are individuals in need, such as ALS patients, intensive care unit patients, confined patients, and patients who lack the ability to communicate verbally. [Brief description of the drawings]

[0049] In order to better understand the subject matter disclosed herein and to illustrate how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Figure 1A] FIG. 1A is a schematic block diagram of a system according to an embodiment of the present disclosure. [Figure 1B] FIG. 1B is a schematic block diagram of a system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic block diagram of a system according to another embodiment of the present disclosure. [Figure 3A] FIG. 3A is a schematic block diagram of a control unit according to one embodiment of the present disclosure. [Figure 3B] FIG. 3B is a schematic block diagram of a control unit according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic visual representation of a menu layer according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a schematic visual representation of a menu layer according to another embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic visual representation of a menu layer according to another embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram of a time-based prompt menu for selection of an item according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a schematic diagram of one embodiment of an eye-tracking based system including a camera, a bone conduction speaker and a control unit. [Figure 9] FIG. 9 is a schematic diagram of one embodiment of joystick-like gesture classification where the location of the pupil region is determined based on a threshold map (innermost square), a position map (middle square) and an ROI map (outermost square). [Figure 10] FIG. 10 is an illustration of one embodiment of the mapping between eye gestures and commands in the single gesture manipulation mode. [Figure 11] FIG. 11 is an illustration of one embodiment of the mapping between eye gestures and commands in two gesture manipulation modes. [Figure 12] FIG. 12 is an illustration of one embodiment of the mapping between eye gestures and commands in the three gesture operation modes. [Figure 13] FIG. 13 is a diagram of one embodiment of the mapping between eye gestures and commands in the four gesture operation modes. [Figure 14]FIG. 14 is a diagram of one embodiment of the mapping between eye gestures and commands in the five gesture operation modes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0050] Reference is first made to Figures 1A and 1B, which show schematic block diagrams of a system according to an embodiment of the present disclosure. The eye-tracking based system 100 comprises a camera 102 mounted in a holder attached to a frame or attached to a user's head. The camera 102 is operable to continuously capture images of one or both of the user's eyes and eyelids and generate image data representative thereof. The system 100 includes a control unit 104 in data communication with the camera 102 and a first output module 106, typically via an actuator module 108 which drives the first output module 106. The output module 106 may be a visual display device, such as a digital screen, or an audible device, such as a speaker, headphones, etc.

[0051] The control unit 104 also includes a processor 110 configured to receive and process image data from the camera 102, identify at least one of a pupil position and an eyelid movement, classify them into gestures including one or more of a pupil position, a sequence of pupil positions, and a sequence of eyelid blinks, and generate gesture data. The processor 110 is also configured to drive a menu generator 112 that drives the presentation of menus to a user through the operation of the actuator module 108, thereby allowing the user to navigate and select menu items by said gesture data.

[0052] 1B shows a block diagram of a system of the present disclosure that interacts with a sensor 115 (e.g., EEG, electromyogram (EMG), or head movement measuring device) for measuring physiological parameters. In particular, the device 115 is in data communication with a control unit 104 that is configured to communicate commands based on detected physiological signals of the user. The physiological signals can be analyzed and translated into commands for the system 100, such as activating the system, starting a search process, selecting a menu item, etc.

[0053] In Figures 2-3, elements that are the same as in Figures 1A and 1B have been given reference numbers shifted by 100. For example, element 204 in Figure 2 performs the same function as element 104 in Figures 1A and 1B. The reader is therefore referred to the description of Figures 1A and 1B to understand their meaning and function.

[0054] The system of FIG. 2 differs from the systems of FIG. 1A and FIG. 1B in that the former also includes a second actuator module 214 operable to drive a second output unit 216 that is part of the system or includes external elements such as warning devices, display screens, utilities for operating devices (such as curtains, music players, lights, etc.) in the user's vicinity. In other words, the second output unit 216 establishes a connection to the environment of the system by a wired (e.g. infrared) or wireless connection to other devices, a connection to a cloud server, or a connection to a means of communication with the user's surroundings. For example, the system can be wirelessly connected, e.g. by Wi-Fi or Bluetooth, to smart home devices that can be operated by gestures of the user using the system 200. The system can be configured to drive the second output unit by certain predefined gestures or through selectable menu items. Such certain gestures can be predefined or user selectable.

[0055] Reference is now made to Figure 3, which shows a schematic block diagram of a control unit according to two aspects of the present disclosure. The control unit 304 has a data entry utility 303 in data communication with a camera 302 (Figure 3A) for capturing successive images of the eye, or with a sensor 301 (Figure 3B) for measuring physiological parameters. Data received by the data entry utility 303 is processed by a processor 310, and the processed data is classified into gestures by a classifier 305. The classified gestures are then transmitted by a data communication module 309 to a computer 307 in data communication with the control unit, thereby controlling the operation of the computer 307.

[0056] Reference is now made to FIG. 4, which is a schematic visual depiction of a menu layer according to an embodiment of the present disclosure. As can be seen, the menu layer has several menu items, each of which is selected by a different gesture. For example, an up gesture UG, i.e., an up position of the pupil, drives the selection to play music. Correspondingly, a left gesture LG drives communication with the caregiver menu, a center gesture CG to watch TV, a right gesture RG to listen to a book, and a down gesture DG to open a free text message menu. Some menu items are enabled by a wireless connection, such as between the TV and the system by Bluetooth or Wi-Fi network, while other menu items, such as listening to a book or playing music, are enabled by a connection to a cloud server. Playing music or listening to a book can be done directly from the cloud server without having to store the data in a local memory. It is noted that data exchange with the cloud works in both ways, downloading data from the cloud to the system and uploading data from the system to the cloud.

[0057] At any time, at any layer of the menu, a user creates a predefined gesture sequence PGS, which triggers a predefined action, such as outputting an emergency alert to a caregiver, e.g., a voice alert via a speaker, a text alert to a mobile device, an alert to a medical center, or a combination thereof. The predefined gesture sequence PGS can be configured according to the user's will, e.g., a sequence of three or four blinks, an up gesture UG, a down gesture DG, a sequence of up gesture UG and down gesture DG, or any other desired sequence.

[0058] Figures 5 - 7 are schematic visual depictions of a menu layer according to another embodiment of the present disclosure, illustrating a unique method of selecting item menus by a free - text method. The characters are clustered into groups of characters, for example, 4, 5, or 6 characters per group. The user can search between groups and select a specific character within a group by performing a specific gesture at an appropriate timing. In Figure 5, the system is displaying a group of characters A, B, C, D, E. The user can create an up - gesture UG to navigate to groups V, W, X, Y, Z or create a down - gesture DG to navigate to groups F, G, H, I, J. Other gestures can be made to trigger other commands such as character deletion, use of the backspace key, returning to the previous menu, which are shown by way of example only. Note that these commands can be replaced with other appropriate commands or deleted. Figure 6 illustrates a user selection of the down - gesture DG in the menu layer of Figure 5 that triggered a menu item including the group of characters F, G, H, I, J. The system can trigger an automatic output session of the characters within the group, such as announcing the name of each character with a time difference from other characters via a speaker or headphones, as illustrated in Figure 7. As can be understood, F is announced at time t1, character G is announced at time t2, etc. When a specific predetermined gesture PG is created, for example, by blinking once or twice, a character is selected. For example, if the predetermined gesture PG is made at time t1 < t < t2, character F is selected, and if the predetermined gesture PG is made at time t3 < t < t4, character H is selected. In another embodiment of the system, the output session of the characters within the group is triggered according to the user's request by the predetermined gesture PG. This is relevant when the subject using the system lacks the ability to perform some gestures such as left, right, up, or down gestures.In this scenario, a search in the system is initiated by a first predefined gesture PG1, and an item menu can be selected by a second predefined gesture PG2, and the first and second predefined gestures can be different or the same. For example, when the system is in the state of Fig. 6, the user can close his eyes to trigger an output session of the letters in the group, and when the desired letter is heard, the user can open his eyes to select the letter. It should be noted that by making an up or down gesture UG, DG, the system navigates to other letter groups, as shown in Fig. 6.

[0059] To improve gesture classification, the system can be trained by machine / deep learning algorithms. First, the system is received with labeled gesture images (blink, center, up, down, right, left) to collect an initial dataset. Then, the system runs a training session with a set of training images. During this training session, the system, i.e., the system's neural network, learns how to recognize each category of labeled images. If the current model makes an error, it corrects and improves itself. Once the network's training session is over, a test set of images is received and processed by the system to check the new classification model. The classifications made by the system are compared with the ground truth labels of the test set and the number of correct classifications is calculated, resulting in precision, recall, and f-measure values ​​used to quantify the performance of such networks.

[0060] A schematic diagram of an eye-tracking-based system for communication support is shown in Figure 8. The system comprises a camera (802) attached to a lightweight head mount (800) (worn on the user's head by a family member, caregiver, or the user himself), bone conduction speakers / headphones (804), and a control unit (not shown).

[0061] Clinical trials conducted by the inventors of the present application demonstrated that patients were comfortable controlling the system after a short test period of a few minutes. As a non-limiting example, in clinical trials conducted at Lambem Hospital in Israel, learning the "call for help" function took an average of 1.12 minutes of training time, learning to communicate a series of pre-determined sentences took an average of 6.44 minutes of training time, and free text character-by-character communication using a mobile screen took an average of 11.08 minutes of training time, as shown in Table 1 below. TIFF0007625673000001.tif49170

[0062] A non-limiting embodiment of the classification of a joystick-like gesture is shown in FIG. 9. The classification is based on finding the location of the pupil region. It is obtained based on the threshold map (innermost square). Specifically, a specific location is determined each time the pupil region touches the boundary or tangent of the threshold map. For example, when the pupil region touches the upper boundary of the threshold map, the image data is classified as an "up" gesture. The threshold map can be derived from the position map (center square), for example, at least 80% away from the center of the position map, and optionally, the position map is within a larger region of interest (ROI) defined based on anatomical features of the eye or its surroundings.

[0063] 10-14 provide diagrams of several embodiments of mapping between eye gestures and commands in single gesture operation mode, 2, 3, 4, and 5 gesture operation modes. According to the mapping shown in FIG. 10, a user initiates a scan session and selects an item by performing a blink gesture. According to the mapping shown in FIG. 11, a user initiates a scan session and selects an item by performing a blink gesture and selects a back command by performing a "right" gesture. According to the mapping shown in FIG. 12, a user traverses menu items with two gestures ("right" and "left") and selects an item by performing a third blink gesture. According to the mapping shown in FIG. 13, a user traverses menu items with three gestures ("right", "left", "up") and selects an item by performing a blink gesture. According to the mapping shown in FIG. 14, a user traverses menu items with four gestures ("right", "left", "up", "down") and selects an item by performing a blink gesture.

Claims

1. 1. A control system for operating a computer, comprising: a camera configured to continuously capture images of a user's eye and / or eyelid and generate image data representative thereof; A first output module; a control unit in data communication with the camera and the computer, (i) receiving and processing the image data to identify at least one of a pupil position and an eyelid movement and classifying them into a gesture comprising one or more of a pupil position, a sequence of pupil positions, and a sequence of eyelid blinks to generate gesture data, the classification including tracking a position of a pupil region, the pupil region including the pupil or any portion of the pupil identified as indicative of a pupil, the position of the pupil region being determined based on a position of the pupil region within a threshold map, and each time the pupil region touches a boundary line of the threshold map or a tangent to a boundary line of the threshold map a specific position is determined, the specific position defining the gesture; (ii) manipulating a hierarchical menu arranged in successive layers of the hierarchy to enable a user to select a first hierarchical menu item and then select a second hierarchical menu item that is lower in the hierarchy than the first hierarchical menu item, the hierarchical menu having hierarchical user-selectable menu items for enabling a user to navigate and select menu items by the gesture data; (iii) a control unit operable to drive the first output module to present the menu items to a user; the first output module is configured to provide an audio presentation of a time-based prompt menu to a user to select an item using a predefined gesture selected from one of the gestures; The first menu item is time t １ and the second menu item is announced at time t ３ The predetermined gesture is announced by t １ <t<t ２ the first menu item is selected when the predetermined gesture is performed at t ３ <t<t ４ the second menu item is selected when 11. The control system of claim 1, wherein processing the image data includes applying machine learning techniques, the machine learning techniques including determining a likelihood that given image data corresponds to a particular gesture by estimating a range of eye movements to classify the gesture, the machine learning techniques including a machine learning algorithm trained with images with labeled gestures.

2. 2. The control system of claim 1, The control unit in data communication with at least one sensor for measuring a physiological parameter; A control system operable to receive and process physiological data acquired by the at least one sensor and classify the physiological data into commands for operating the computer.

3. 3. The control system according to claim 1, The control unit is in data communication with at least one sensor for measuring at least one physiological parameter, the control unit further comprising: receiving and processing at least one physiological data and classifying it into at least one command, said at least one command comprising any body part movement, breathing pattern, smell, sound output, change in muscle tone, skin conductance, nerve output, or any combination thereof; A control system that operates the computer using the command.

4. 4. The control system of claim 3, wherein the sensor is an electroencephalography (EEG) device.

5. A system according to any one of claims 1 to 4, characterized in that the first output module enables a user to communicate with other individuals.

6. A system according to any one of the preceding claims, characterized in that the gestures include centre, right, left, top and bottom positions of the eye pupil.

7. A system according to any one of the preceding claims, characterized in that the gesture comprises a sequence of two or more eyelid blinks.

8. A system according to any one of the preceding claims, characterized in that the computer is arranged to drive the first output module to output an alarm.

9. a control unit configured in data communication with a camera that captures successive images of the eye and a computer to generate image data representative of the successive images of the eye, The control unit: (i) receiving and processing the image data to identify at least one of a pupil position and an eyelid movement and classifying them into a gesture comprising one or more of a pupil position, a sequence of pupil positions, and a sequence of eyelid blinks to generate gesture data, the classification including tracking a position of a pupil region, the pupil region including the pupil or any portion of the pupil identified as indicative of a pupil, the position of the pupil region being determined based on a position of the pupil region within a threshold map, and each time the pupil region touches a boundary line of the threshold map or a tangent to a boundary line of the threshold map a specific position is determined, the specific position defining the gesture; (ii) manipulating a hierarchical menu arranged in successive layers of the hierarchy to enable a user to select a first hierarchical menu item and then select a second hierarchical menu item that is lower in the hierarchy than the first hierarchical menu item, the hierarchical menu having hierarchical user-selectable menu items for enabling a user to navigate and select menu items by the gesture data; (iii) operable to drive a first output module to provide an audio presentation of a time-based prompt menu to a user for selecting an item using a predefined gesture selected from one of said gestures; The first menu item is time t １ and the second menu item is announced at time t ３ The predetermined gesture is announced by t １ <t<t ２ the first menu item is selected when the predetermined gesture is performed at t ３ <t<t ４ the second menu item is selected when 11. The control system of claim 1, wherein processing the image data includes applying machine learning techniques, the machine learning techniques including determining a likelihood that given image data corresponds to a particular gesture by estimating a range of eye movements to classify the gesture, the machine learning techniques including a machine learning algorithm trained with images with labeled gestures.

10. 10. The control unit according to claim 9, the control unit being in data communication with at least one sensor for measuring a physiological parameter; The control unit receives and processes physiological data acquired by the sensors, classifies them into commands, and controls the operation of the computer by sending corresponding commands to the computer.

11. 1. A control system for operating a computer, comprising: a camera mounted in a holder attachable to a head of a user and configured to continuously capture images of one or both of the user's eyes and eyelids and generate image data representative thereof; A first output module; a control unit in data communication with the camera and the computer, (i) receiving and processing the image data to identify at least one of a pupil position and an eyelid movement and classifying them into a gesture comprising one or more of a pupil position, a sequence of pupil positions, and a sequence of eyelid blinks to generate gesture data, the classification including tracking a position of a pupil region, the pupil region including the pupil or any portion of the pupil identified as indicative of a pupil, the position of the pupil region being determined based on a position of the pupil region within a threshold map, and each time the pupil region touches a boundary line of the threshold map or a tangent to a boundary line of the threshold map a specific position is determined, the specific position defining the gesture; (ii) manipulating a hierarchical menu arranged in successive layers of the hierarchy to enable a user to select a first hierarchical menu item and then select a second hierarchical menu item that is lower in the hierarchy than the first hierarchical menu item, the hierarchical menu having hierarchical user-selectable menu items for enabling a user to navigate and select menu items by the gesture data; (iii) a control unit operable to drive the first output module to present the menu items to a user; the first output module is configured to provide an audio presentation of a time-based prompt menu to a user to select an item using a predefined gesture selected from one of the gestures; The first menu item is time t １ and the second menu item is announced at time t ３ The predetermined gesture is announced by t １ <t<t ２ the first menu item is selected when the predetermined gesture is performed at t ３ <t<t ４ the second menu item is selected when 11. The control system of claim 1, wherein processing the image data includes applying machine learning techniques, the machine learning techniques including determining a likelihood that given image data corresponds to a particular gesture by estimating a range of eye movements to classify the gesture, the machine learning techniques including a machine learning algorithm trained with images with labeled gestures.

Images