Using gaze-based communications in medical assessment

Abstract

A closed eye of a patient is imaged using an eyetracker. A request is received to receive notification when the closed eye is opened from one or more operator input devices. The eyetracker is instructed to determine when the eye is opened. A notification is received from the eyetracker that the eye has been opened. The notification is displayed on one or more operator displays. If the patient is nonverbal, the eye is imaged using the eyetracker. One or more patient selection screen options are displayed on an operator display. A selected option is received from the input devices. A patient selection screen is displayed on one or more patient displays based on the selected option. The eyetracker is instructed to determine a selection by the patient by monitoring the fixation of the eye. The selection is received from the eyetracker and displayed on the operator display.

Description

USING GAZE-BASED COMMUNICATIONS IN MEDICAL ASSESSMENTRELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 688,253, filed on August 28, 2024. the content of which is incorporated by reference herein in its entirety.FIELD OF THE INVENTION

[0002] The teachings herein relate to a system for communicating with a nonverbal patient in a medical assessment setting. More particularly, the teachings herein relate to systems and methods for communicating with a patient if they are unable to speak.BACKGROUND

[0003] Effective patient communications are crucial for optimal medical outcomes.When a patient is nonverbal, poor communication can cause missed or erroneous diagnoses, leading to incorrect, insufficient, or excessive treatment. Nonoptimal medical outcomes eventually result in slower recovery times, costly extended hospital stays, and reduced customer satisfaction.

[0004] Specifically, medical staff often need to communicate with nonverbal patients who are groggy, confused, in pain, stressed, scared, delirious, or intubated. This is particularly true in the intensive care unit (ICU), where patients are often coming out of surgical anesthesia.

[0005] Typically, a patient monitoring system in an ICU can include several connected devices for monitoring, for example, vital signs, cardiac function, respiratoryfunction, and neurological function. However, specific devices to enhance communication between medical staff and nonverbal patients remain lacking.

[0006] As a result, there is an unmet need for systems and methods that can enhance communication between medical staff and nonverbal patients in medical assessment settings such as hospital ICUs, emergency rooms and stroke units, and in on-scene and ambulance emergency situations.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The skilled artisan will understand that the drawings, described below, are for illustration purposes only. The drawings are not intended to limit the scope of the present teachings in any way.

[0008] Figure 1 is a block diagram that illustrates a computer system, upon which embodiments of the present teachings may be implemented.

[0009] Figure 2 is an exemplary diagram of an eyetracking communication system, in accordance with various embodiments.

[0010] Figure 3 is an exemplary diagram of a full, frontal view of a human body that is displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0011] Figure 4 is an exemplary diagram illustrating how PainPoinf s virtual camera moves from its original, front-body-centered position to a final, rear-lower-leg position as a patient looks toward his right calf, upon which embodiments of the present teachings may be implemented.

[0012] Figure 5 is an exemplary diagram of "yes’ and ‘no?selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0013] Figure 6 is an exemplary diagram of a frozen body image with ‘yes’ and ‘no’ selections displayed below on a display of the eyetracking communication system, in accordance with various embodiments.

[0014] Figure 7 is an exemplary diagram of numerical pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0015] Figure 8 is an exemplary diagram of emoji pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0016] Figure 9 is an exemplary diagram of text pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0017] Figure 10 is an exemplary diagram of text pain type selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0018] Figure 11 is an exemplary diagram of the version of the Vanderbilt delirium test selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0019] Figure 12 is an exemplary diagram of a version of the bed control selections displayed on a display of the eyetracking communication system, in accordance with vanous embodiments.

[0020] Figure 13 is an exemplary diagram of keyboard selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0021] Figure 14 is an exemplary' diagram showing five positions of a calibration dot as it moves around a screen, in accordance with various embodiments.

[0022] Figure 15 is an exemplary7flowchart showing a method for communicating with a nonverbal patient, in accordance with various embodiments.

[0023] Figure 16 is an exemplary7flowchart showing a method for monitoring the sleep of a patient, in accordance with various embodiments.

[0024] Before one or more embodiments of the present teachings are described in detail, one skilled in the art will appreciate that the present teachings are not limited in their application to the details of construction, the arrangements of components, and the arrangement of steps set forth in the following detailed description or illustrated in the drawings. Also, it is to be understood that the phraseology7and terminology used herein is for the purpose of description and should not be regarded as limiting.DESCRIPTION OF VARIOUS EMBODIMENTSCOMPUTER-IMPLEMENTED SYSTEM

[0025] Figure 1 is a block diagram that illustrates a computer system 100, upon which embodiments of the present teachings may be implemented. Computer system 100 includes a bus 102 or other communication mechanism for communicating information, and a processor 104 coupled with bus 102 for processing information. Computer system 100 also includes a memory 106, which can be a random-access memory (RAM) or other dynamic storage device, coupled to bus 102 for storing instructions to be executed by processor 104. Memory 106 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 104. Computersystem 100 further includes a read only memory (ROM) 108 or other static storage device coupled to bus 102 for storing static information and instructions for processor 104. A storage device 110, such as a magnetic disk or optical disk, is provided and coupled to bus 102 for storing information and instructions.

[0026] Computer system 100 may be coupled via bus 102 to a display 112, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. An input device 114, including alphanumeric and other keys, is coupled to bus 102 for communicating information and command selections to processor 104. Another type of user input device is cursor control 116, such as a mouse, a trackball or cursor direction keys for communicating direction information and command selections to processor 104 and for controlling cursor movement on display 112.

[0027] A computer system 100 can perform the present teachings. Consistent with certain implementations of the present teachings, results are provided by computer system 100 in response to processor 104 executing one or more sequences of one or more instructions contained in memory 106. Such instructions may be read into memory 106 from another computer-readable medium, such as storage device 110. Execution of the sequences of instructions contained in memory 106 causes processor 104 to perform the process described herein.

[0028] Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present teachings. For example, the present teachings may also be implemented with programmable artificial intelligence (Al) chips with only the encoder neural network programmed - to allow for performance and decreased cost. Thus, implementations of the presentteachings are not limited to any specific combination of hardw are circuitry and software.

[0029] The term “computer-readable medium” or “computer program product” as used herein refers to any media that participates in providing instructions to processor 104 for execution. The terms “computer-readable medium” and “computer program product” are used interchangeably throughout this written description. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 110. Volatile media includes dynamic memory, such as memory 106.

[0030] Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD- ROM, digital video disc (DVD), a Blu-ray Disc, any other optical medium, a thumb drive, a memory' card, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.

[0031] Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to processor 104 for execution. For example, the instructions may initially be carried on the magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 100 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector coupled to bus 102 can receive the data carried in the infra-red signal and place the data on bus 102. Bus 102 carries the data to memory 106,from which processor 104 retrieves and executes the instructions. The instructions received by memory 106 may optionally be stored on storage device 110 either before or after execution by processor 104.

[0032] In accordance with various embodiments, instructions configured to be executed by a processor to perform a method are stored on a computer-readable medium. The computer-readable medium can be a device that stores digital information. The computer-readable medium is accessed by a processor suitable for executing instructions configured to be executed.

[0033] The following descriptions of various implementations of the present teachings have been presented for purposes of illustration and description. It is not exhaustive and does not limit the present teachings to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practicing of the present teachings. Additionally, the described implementation includes software but the present teachings may be implemented as a combination of hardware and software or in hardware alone. The present teachings may be implemented with both object-oriented and non-object-oriented programming systems.EYETRACKING COMMUNICATION SYSTEM

[0034] As described above, effective patient communications are crucial for optimal medical outcomes. Medical staff often need to communicate with nonverbal patients who are groggy, confused, in pain, stressed, scared, delirious, orintubated. This is particularly true in the intensive care unit (ICU), where patients are often coming out of surgical anesthesia.

[0035] Typically, a patient monitoring system in an ICU can include several connected devices for monitoring, for example, vital signs, cardiac function, respiratory function, and neurological function. However, specific devices to enhance communication between medical staff and nonverbal patients remain lacking.

[0036] As a result, there is an unmet need for systems and methods that can enhance communication between medical staff and nonverbal patients in hospital settings such as the ICU.

[0037] In various embodiments, an eyetracking communication system including one or processors, one or more displays, and at least one eyetracker, allows a person’s natural eyegaze behavior to provide a viable communication alternative to their voice when their speech is limited or not available.

[0038] Figure 2 is an exemplary diagram 200 of an eyetracking communication system, in accordance with various embodiments. In Figure 2, the eyetracking communication system includes one or more processors 210, one or more operator displays 220, one or more operator input devices 230, one or more patient displays 240, and at least one eyetracker 250.

[0039] One or more processors 210 are used by the medical staff. One or more processors 210 can be, but are not limited to, tablet computers, smartphones, laptop computers, computer systems as depicted by Figure 1, or any devices capable of sending and receiving control signals or data and analyzing data. One or more processors 210 can include, for example, communications port 215 for communicating with at least one eyetracker 250. peripheral devices, and a patient monitoring system of the ICU. Communications port 215 can connect to one ormore wired or wireless channels and can communicate using wired or wireless communication protocols.

[0040] One or more operator displays 220 are used by the medical staff to receive output from the eyetracking communication system. One or more displays 220 can include any type of display technology' including, but not limited to, cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), or organic light-emitting diode (OLED).

[0041] One or more operator input devices 230 are used by the medical staff to provide input to the eyetracking communication system. One or more operator input devices 230 can include any type of input device including, but not limited to, a touchscreen, a keyboard 231, a mouse 232, a camera 233, or a microphone 234.

[0042] One or more patient displays 240 are placed in front of a nonverbal patient so that the nonverbal patient can see what is depicted on one or more displays 240. One or more displays 240 can include any type of display technology including, but not limited to, cathode ray tube (CRT), liquid crystal display (LCD), lightemitting diode (LED), or organic light-emitting diode (OLED).

[0043] At least one eyetracker 250 is an eyetracking system such as the Eyegaze Edge® sold by Eyegaze Inc. of Fairfax, VA. At least one eyetracker 250 includes at least one camera for imaging the eye of the nonverbal patient. At least one eyetracker 250 can also include a gimbal or electromechanical arm for automatically following the movement of an eye of the nonverbal patient. At least one eyetracker 250 can also include a speech synthesizer and speaker that provides audio feedback to a patient based on their eye movements. At least one eyetracker 250 can include its own processor and display for displaying information to the nonverbal patient. In other words, one or more processors 210 and one or morepatient displays 240 can be part of at least one eyetracker 250 in various embodiments.

[0044] In various embodiments, the eyetracking communication system further includes mobile stand 260 for moving the eyetracking communication system close to the patient.

[0045] For example, for rapid, optimum surgical outcomes, it is important for the nurse to establish accurate and reliable communications with the patient as quickly as possible once the patient awakens from anesthesia. The eyetracking communication system of Figure 2 enhances the patient’s ability to communicate with his nurse, despite his being groggy, confused, in pain, or possibly delirious or intubated.

[0046] In various embodiments, to facilitate different topic areas of patient / nurse communications, the eyetracking communication system has a set of applications (apps), where each app supports a specific topic area. As the nurse wishes to switch between topics, she selects (clicks) her desired app from a menu on her handheld touchscreen device. The nurse display also shows a copy of the patient display, so she does not have to look at the patient’s screen to see what he sees.Sleep App

[0047] The first role the eyetracking communication system plays with a patient occurs when he is still unconscious and is moved into the recovery room after surgery. The eyetracking communication system’s Sleep app helps the nursing staff monitor when the patient wakes up from anesthesia.

[0048] After the patient’s bed is positioned on the recovery room floor, the nurse rolls the eyetracking communication system up over the bed and positions the patient’s oneor more displays above the patient's chest and head - where the eyetracking camera has a clear, direct view of the patient’s face and eyes. (As shown in Figure 2, the one or more displays for the patient can be mounted on a mobile or rolling stand that has an adjustable pivot mount that makes it easy for the nurse to position a screen in front of the patient’s face, independent of how he is lying in bed.)

[0049] While the patient remains unconscious, the display screen remains turned off, where it presents him with minimal visual stimulation as long as he’s unconscious. At the same time, the eyetracking communication system’s eyetracking camera is active. The Sleep app passively observes the patient’s eyes, detects when his eyes open, and, if a nurse is not present at the time, remotely notifies the staff of the patient’s revival of consciousness.

[0050] The key purpose of the Sleep app is to minimize the time the patient is awake before he receives direct nurse attention. When a patient first wakes up, immediate nurse intervention minimizes any negative psychological / emotional reaction he may have to his new, confused, painful, delirious, or intubated state.

[0051] In various embodiments, the negative psychological / emotional reaction a patient may have when awakening in the ICU is further ameliorated by the eyetracking communication system. In addition to notifying staff that the patient is awake, the detection of eye opening can trigger the eyetracking communication system to display images or video on one or more patient displays 240 meant to calm the patient. The images or video can include, but are not limited to. images or video of scenery, loved ones or friends, or the attending staff. Images of video of loved ones, friends, or staff can be recorded or can be provided in real-time using camera 233, for example. In still other embodiments, images of loved ones,friends, or staff are animated using artificial intelligence to help calm the patient and to provide instruction on how to use the eyetracking communication system.

[0052] In various embodiments, the detection of eye opening can also trigger monitoring by a device of the patient monitoring system of a responsible medical organization. For example, the detection of eye opening can trigger neurological function monitoring. Specifically, one or more processors 210 further send a notification to a patient monitoring system of a responsible medical organization to begin neurological function monitoring upon detection of eye opening using communications port 215.

[0053] When the nurse initially communicates with a patient as he regains consciousness, her first objective is to orient him to his current situation and assure him of his safety. To do this, she rolls the eyetracking communication system’s screen away, does her usual patient-wake-up routine - making direct face-to-face contact, telling him he’s in the hospital, assuring him he’s safe, and beginning to make her initial assessment of the state of his evolving cognizance and / or delirium.

[0054] The eyetracking communication system provides a novel, alternative means for the patient to communicate - based on his eye activity rather than his standard verbal speech. Similar to graphical user interfaces (GUIs) in modem computer technology (where the user manually performs a mouse click to activate an icon), the eyetracking communication system presents graphic displays to the patient who, in place of using a mouse or touch screen, looks at different icons and objects to actuate the desired computer activity - and ultimately convey information to the nurse.

[0055] A key challenge to using the eyetracking communication system in an ICU. however, is introducing a gaze-based communication methodology to a patientwho wakes up groggy', confused, in pain, delirious, or intubated. The last thing he wants to do at this point is learn a new computer technology. Thus, initial patient / nurse communications with the eyetracking communication system are specially designed to operate without any prior patient knowledge of or experience with eyetracking technology'.

[0056] If the nurse decides that the eyetracking communication system might be a useful communication aid, she begins by telling the patient that she is going to use a video display to help with the communications. She turns the screen display on and rolls it back into position where the patient can easily see his screen.

[0057] Typically, the first specific information the nurse wants to get from the patient is his state of pain, which, generally, is also the first thing the patient wants to complain about. To help him designate exactly where on his body he is experiencing pain, the nurse switches the eyetracking communication system into the PainPoint app.PainPoint App

[0058] The PainPoint app begins by displaying a full, frontal view of a human body. The body is the only object in the display, and it fills the Patient Screen. As the bodyimage display appears, the nurse explains, ‘‘This picture display will help us communicate. Just look at it. You don’t have to talk.”

[0059] Figure 3 is an exemplary diagram 300 of a full, frontal view of a human body that is displayed on a display of the eyetracking communication system, in accordance with various embodiments. The full, frontal view of a human body is displayed by the PainPoint app.

[0060] Initially, the patient naturally glances around the image to get a sense of the scene he’s looking at. The nurse then instructs: “Now. please just look where you hurt most. I’ll be able to see exactly where you are looking."’ Given that a) the patient is likely thinking about his pain, b) the nurse has verbally indicated that she wants to talk about pain, and c) a picture of a body is displayed in front of him, the patient is naturally incentivized to look directly at his pain point - despite grogginess, confusion, or even delirium.

[0061] During PainPoinfs initial interaction, the eyetracker watches and tracks where on the body the patient looks. At some point, the patient begins to visually concentrate on a particular area of the body. When the eyetracker detects the patient’s focus on a particular body region, PainPoint assumes his designated pain point is in that area, and it begins to automatically adjust the body display to make it easier for the patient to see more detail: It gradually begins to zoom in on that region of the body and move the display of that region to the center of the screen.

[0062] As the display of the human body zooms and shifts, the nurse can see this on her copy of the body image shown on her handheld device. For the nurse, PainPoint also displays a ’+’ mark at the patient’s visually designated pain point.

[0063] The location of the '+’ mark on the body begins to answer the nurse’s question. She might say something like, “I can see you’re looking down toward your right leg. If that’s where it hurts, just keep looking exactly at your pain point.” PainPoint continues zooming and shifting until a fairly magnified view of the visually designated body point has moved to the center of the screen, where the pain point is directly visible to both the patient and nurse.

[0064] In addition to PainPoinfs zooming and shifting the body image in response to his gaze activity, the app automatically rotates the angular point of view of the bodyto provide a direct (straight-on) perspective of the body surface at the pain point. This feature allows the patient to designate a pain point on any side of his bodyfront, back, right, left, top, or bottom - while continuously viewing the image of a single body.

[0065] Figure 4 is an exemplary7diagram 400 illustrating how PainPomt's virtual camera moves from its original, front-body-centered position to a final, rear-lower-leg position as a patient looks toward his right calf, upon which embodiments of the present teachings may be implemented. The body images displayed at the start and end of the process are shown at the bottom of Figure 4. This movement of the virtual camera is described in U.S. Patent 11,849,098, which is incorporated herein by reference.Communication Verify7App

[0066] Effective patient / nurse communication requires the nurse to obtain high confidence in their interpretation of the patient’s answers. In standard practice, a nurse ty pically verifies information by repeating an answer back to the patient and asking him to confirm it.

[0067] In various embodiments, using the eyetracking communication system, the verification process is done with yes / no questions that the patient answers visually simply by looking at a pair of ‘yes’ and ‘no’ keys displayed on his screen. He visually fixates on the key representing his answer. The use of eyetracking has previously been described for use in communicating with locked-in patients in U.S. Patent 9,244,528, which is incorporated herein by reference.

[0068] Figure 5 is an exemplary diagram 500 of ‘yes’ and ‘no’ selections displayed on a display of the eyetracking communication system, in accordance with variousembodiments. The ‘yes’ and ‘no’ selections are displayed by the CommunicationsVerify app.

[0069] When using the PainPoint app, the nurse uses the following procedure to verify that the designated pain point is correct. First, she freezes the motion of the body image. This frees the patient to look around the screen (or even off the screen, at the nurse for example) without inducing further shifting of the body display. Then the nurse might say, “I’ve just frozen the image. You can now look around freely without causing the body image to move any more. Now, let’s verify that this is actually your pain point.”

[0070] The nurse then activates PainPoinfs Verify feature, that: places a prominent ‘+’ mark at the expected pain point on the patient’s now frozen body image, and reduces the size of the frozen body display to make room for a ‘yes-no’ verification display at the bottom of the patient display.

[0071] Figure 6 is an exemplary diagram 600 of a frozen body image with 'yes’ and 'no’ selections displayed below on a display of the eyetracking communication system, in accordance with various embodiments. The changes shown in Figure 6 in the screen display trigger several natural, yet typically unconscious, responses in the patient’s brain. He naturally looks all around the screen to see what is different. He sees that yes / no buttons have suddenly appeared, and he may initially wonder what they are all about. He notices that the body display has shrunk. And he sees that a '+’ mark has appeared on it. These natural responses result in behavior thatsuccessfully communicates information to his nurse, even though he may not yet initially realize that he is actually communicating.

[0072] After the nurse gives the patient some time to process the changes he's seeing, she verbally asks, “Does the ‘+’ mark on the body image match your pain point?” He considers whether the ‘+’ mark matches his pain point.

[0073] Finally, the nurse "enables' the yes / no keys (which signals the eyetracker to begin to respond to patient gazes at the keys), and she asks, “Please answer by looking at the ‘yes’ or ‘no’ button at the bottom of the screen.”

[0074] ‘Enabling’ the yes / no keys causes the Verify app to actively respond when the patient looks directly at either of the keys. When the patient looks at one of the keys, Verify" s response consists of two actions: the ‘yes’ or ’no’ box in the display briefly flashes, visually indicating to the patient that he has ‘clicked’ that key, and a speech synthesizer in the eyetracking communication system verbalizes the appropriate ‘yes’ or ‘no’ keyword, speaking out loud for both the patient and nurse to hear.

[0075] When the PainPoint app performs a verification, the eyetracking communication system assumes that gazes within the (frozen and reduced) body -image portion of the display indicate that the patient is still in the process of thinking about his answer (either consciously or unconsciously), and it ignores those gazes.

[0076] It is only when the patient’s gaze goes into one of the yes / no keys that the Verify begins to hypothesize that the patient may be signaling his actual answer. For the system to ensure that the patient is intentionally looking at a key, he must hold his gaze within the key for a significant period of time (typically about half a second) before it triggers its verbal response. Verify does not respond if the gaze simply happens to pass quickly through a key.PainLevel and PainTvpe Apps

[0077] In various embodiments, the eyetracking communication system displays pain level information to a nonverbal patient to obtain a pain level selection from the patient. The patient visually fixates on the area of the display representing his pain level.

[0078] Figure 7 is an exemplary diagram 700 of numerical pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0079] Figure 8 is an exemplary diagram 800 of emoji pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0080] Figure 9 is an exemplary diagram 900 of text pain level selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.

[0081] In various embodiments, the eyetracking communication system displays pain type information to a nonverbal patient to obtain a pain level selection from the patient. The patient visually fixates on the area of the display representing his pain type.

[0082] Figure 10 is an exemplary diagram 1000 of text pain type selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.Delirium App

[0083] In various embodiments, the eyetracking communication system displays a version of the Vanderbilt delirium test to a nonverbal patient. The patient uses the displayed test to make a selection in response to a nurse request to “hold up one more finger than I am.” The patient visually fixates on number of fingers they would hold up.

[0084] Figure 11 is an exemplary diagram 1100 of the version of the Vanderbilt delirium test selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.BedControl App

[0085] For ICU patients who are uncomfortable moving, it is often more desirable to operate their bed controls with their eyes rather than their hands. In various embodiments, the eyetracking communication system displays bed controls to a nonverbal patient. The patient visually fixates on the area of the display representing his preferred bed control.

[0086] Figure 12 is an exemplary diagram 1200 of a version of the bed control selections displayed on a display of the eyetracking communication system, in accordance with various embodiments.CallButton App

[0087] The CallButton app allows the patient to "press’ his call bell by looking at a visual call button displayed on his screen - without having to physically find the device and press the button manually.Patient's In-situ Learning to Use His Eyes to Communicate

[0088] Since few ICU patients have any prior knowledge or experience in communicating with their eyes, a critical objective of the eyetracking communication system is to make the system produce immediate communication results with minimum patient training.

[0089] Though the Sleep app may be the first app the patient uses to “communicate” to the nurses that he’s waking up, the act of opening his eyes is invol untary and does not constitute deliberate communication. The patient does not leam about intentional gaze-based communication from his experience of waking up. His first real eyegaze learning experience begins with PainPoint, the first app used after Sleep.

[0090] PainPoint is designed with two key operational objectives: provide the nurse with critical medical information about the patient, and begin to give the patient basic training in the art of deliberate, gaze-based communication.

[0091] As seen above with regard to the PainPoint app, the nurse guides the patient’s initial experience with a scripted approach designed to help the patient quickly leam both the operational method and intrinsic value of using his eyes to communicate. Along with the use of the visual body display, which the brain’s visual cortex naturally and immediately recognizes (unless he has severe delirium), the nurse explains that the patient will use his eyes to communicate. She coaches him to simply look at his pain point. She explicitly states that he does not have to talk. As the patient sees the body image move, he begins to get the experience that the system can tell what he’s looking at; that the system is actively responding to what it is looking at; and that the automated response gives him a better view of what he’s looking at. He also hears the nurse telling himwhat she's seeing and that she's getting the answer to her question. All these conditions facilitate rapid learning of the method and value of gaze-based communications.

[0092] By the time the nurse freezes the image, the yes / no confirmation buttons appear, and the nurse asks if the ‘+’ accurately marks his pain spot, the patient is ty pically becoming consciously aware that he is communicating valuable information with his eyes - without having to talk. Ideally, the patient’s PainPoint experience produces a positive psychological / emotional response that motivates him to continue learning more advanced eyegaze communication methods.Ev eTvping App

[0093] A highly useful eyegaze communication method for patients in the ICU is eyetyping. While the PainPoint, PainLevel, PainType, Delirium and C allButton apps enable the communication of critical medical information, each of their informational scopes are limited to narrow topic areas. With eyetyping, the patient can say anything he wishes - without limitation to specific conversational topics chosen by the nurse. He is free to contribute directly to the planning of his recovery plan.

[0094] To “type with your eyes ” a keyboard is displayed on the screen and the patient types by fixating his gaze on the sequence of characters - rather than activating them with his fingers. As the patient types, his typed text appears in the area above the keyboard, for both him and the nurse to see. A special speak key (SPK) on the gaze-based keyboard causes the eyetracking communication system’s speech synthesizer to speak the words he’s just typed with his eyes.

[0095] Figure 13 is an exemplary' diagram 1300 of keyboard selections displayed on a display of the eyetracking communication system, in accordance with various embodiments. The patient visually fixates on an area of the display to ty pe the selection.

[0096] Eyetyping is an advanced form of the yes / no buttons used in the Verify app. In both cases the patient activates a key by fixating his gaze on it for a specified period of time, called the dwell time. For a new eyegaze user, such as an ICU patient, the dwell time is ty pically half a second.

[0097] If, after running the earlier apps, the nurse suspects the patient appreciates the eyetracking communication system’s communication value and may be motivated and able to ty pe with his eyes, she can ask him, “Would you like to speak for yourself and type out your own words?” If he answers yes, she switches his screen into EyeTyping app.Eve Calibration

[0098] A key difference between the eyetracking communication system’s Verify and EyeTyping apps is that Verify has two keys, while EyeTyping has about 30. This means that the EyeTyping keys must be significantly smaller to fit them all on the screen. Consequently, the eyetracker has to measure the patient’s gazepoint more accurately to correctly determine which specific EyeTyping key he is looking at.

[0099] The ocular muscles in the human eye are able to point the eyes with very high precision. However, while the eyes themselves point accurately, eyetrackers have a problem measuring the gaze accurately - unless the user performs an eyetracking calibration procedure, run by the Calibration app.

[0100] The need for calibration arises because the physiological properties of individual eyeballs (such as the cornea radius of curvature and the offset of macular region from the eye’s optic axis) vary from one eye to another across the human population. Accurate gaze prediction depends on the eyetracker having accurate values for these anatomical eyeball parameters.

[0101] The EyeCalibration procedure obtains accurate eye-parameter values by having the user visually follow a dot as it moves successively to various points around the screen. Since the Calibration app knows where on the screen it displays each calibration dot (and thus it knows where the patient is actually looking), it optimizes the values of its physiological eye parameters to make the predicted gazepoints match the true calibration points.

[0102] Figure 14 is an exemplary diagram 1400 showing five positions of a calibration dot as it moves around a screen, in accordance with various embodiments.

[0103] Note: During the patient’s initial operation of the earlier non-eyetyping apps, he has not yet calibrated the eyetracker. While uncalibrated, the eyetracker assumes the eyes are “standard,” i.e. their anatomical parameter values are set to average values over the full human population. Since the standard-eye values produce sufficient eyetracking accuracy to reliably determine when people are looking at the large keys used in the eyetracking communication system’s non-typing apps, the patient does not need to calibrate until they want to use the EyeTyping app.

[0104] In summany a key objective of the medical assessment eyetracking communication system’s design is to allow a patient who has no knowledge or experience with eyetracking to effectively use the technology immediately upon waking up after surgery.

[0105] Today’s gaze-based communication devices generally employ visual button pushing methods to implement eye typing, web surfing, telephone communication, home control and computer operation. While these devices provide broad functionality, users need significant operational experience before they realize the benefits. They must: have significant pre-training, develop an explicit understanding of eyetracking, become familiar with the complex screen layouts, and have a good set of eye calibration data.

[0106] The eyetracking communication system implements a carefully designed introductory experience without requiring such prior experience or training. PainPoint simultaneously: allows the patient to answer essential medical questions immediately and subtly introduces the patient to eyetracking technology without him initially even being aware that his eyes are being tracked.Engagement of Broad Mental Functions

[0107] To achieve these goals, the PainPoint application is designed to stimulate a broad set of preexisting, intuitive brain functions that evoke natural eye behaviors that, in turn, result in the patient naturally answering the pain point question with minimum cognitive learning stress. With its graphical display of a moving human body, the PainPoint app naturally engages the brain’s visual cortex, which automatically triggers and utilizes the patient’s preexisting knowledge of human body anatomy. The visual cortex then passes this processed visual data on too many downstream parts of the brain, in particular, higher-level functions generally located in the cerebral cortex.

[0108] As the nurse speaks, the Wernicke section of the patient’s brain interprets her words and, similar to the visual cortex, passes the verbal information out to otherbrain functions. The high-level brain functions decide what’s important to the person at the time, formulate a strategy for dealing with these issues, and drive the body's motor functions to implement these strategies.

[0109] One of the brain’s key motor functions is controlling its eyes. The superior colliculus gathers information from all over the brain, continuously establishes the most important thing in the outside world for the eyes to look at any moment, and directs the eyes to point there. In turn, the visual cortex gets a view of this most important thing and passes the resulting visual information on for the brain to decide what to do and look at next.

[0110] The brain’s eye / visual-cortex / frontal-lobe / superior-caliculus processing path embodies a central, continuous brain / body feedback loop that is fundamental to our life operation. A key objective of the eyetracking communication system is to stimulate the operation of this brain function loop, via the visual graphical display and the auditory nurse input, to help focus the patient’s attention on his medical communication task.[0011 1] Ideally, the eyetracking communication system’s exercise of the patient’s eye / brain feedback loop also speeds the brain’s awakening from its anesthetized state, accelerating its return to full consciousness and reducing the duration of any delirium.Embedded Training Pedagogy

[0112] As discussed above, the underlying goal of the patient / nurse communication process is to subtly implement an embedded pedagogy for teaching eyetracking communication skills.

[0113] The patient’s implicit interest in his own pain, coupled with the brain's intrinsic behavior to point its eyes at what’s most important to it at any time, naturally, even if unconsciously, motivates the patient to look directly at his pain point. PainPoinfs zoom feature enables the patient to specify a relatively precise point despite low accuracy eyetracking that may result from not performing an eye calibration procedure. When the nurse verbally acknowledges that she’s ‘heard’ the patient’s pain point answer, the patient begins to consciously perceive that he is carrying on a conversation - he’s conveyed information via his gaze activity - without his having to say anything. The success of the patient’s gaze-based communication motivates him to continue to use his eyes in that endeavor.

[0114] To reinforce this initial indirect eyetracker training, the nurse can later explicitly explain to him, ‘‘The system uses a video camera to watch your eyes and figure out what you were looking at. I was watching a copy of your screen, so I knew what you were looking at. You essentially used your eyes to tell me wiiere your pain was. You did not have to say a word. You ran the program with your eyes. This technology is called ‘eyetracking.’ Now you can begin to understand how people communicate with their eyes.” At this point, having had a successful eyetracking experience and having heard an explicit explanation of what he’s just done, he’s likely willing and motivated to continue learning more.

[0115] For people who cannot speak, eyetracking provides an option to utilize their powerful eye / brain feedback loop to augment their communication functions. In medical assessments, such as in the ICU, the design of the eyetracking communication system provides a targeted means to execute the specific patient / nurse communications needed to optimize the speed and efficiency of the surgical recovery' process. The eyetracking communication system's usage scenario both enables critical communication processes and implicitly teaches the patient a new way to use their eyes to stand in for their voice when they cannot speak.System for communicating with a nonverbal patient

[0116] Figure 2 depicts a system for communicating with a nonverbal patient, in accordance with various embodiments. The system includes at least one eyetracker 250 that images an eye of a patient, one or more patient displays 240, one or more operator displays 220, one or more operator input devices 230, and

[0117] one or more processors 210.

[0118] One or more processors 210 display one or more patient selection screen options on one or more operator displays 220. One or more processors 210 receive a patient selection screen option from one or more operator input devices 230. One or more processors 210 display a patient selection screen on one or more patient displays 240 based on the received patient selection screen option. One or more processors 210 instruct at least one eyetracker 250 to determine a selection of the patient selection screen by the patient by monitoring a fixation of the eye of the patient on the patient selection screen. One or more processors 210 receive theselection from at least one eyetracker 250. Finally, one or more processors 210 display the selection on one or more operator displays 220.

[0119] In various embodiments, the system also monitors the commands of the medical staff operator and does start monitoring for a patient selection until the command is given. The command can be a word, phrase, or sentence, for example. Specifically, one or more input devices 230 include operator microphone 234. One or more processors 210 further monitor operator microphone 234 for a verbal command before instructing at least one eyetracker 250 to determine a selection of the patient selection screen by the patient.

[0120] In various embodiments, all patient selections are also sent to the patient monitoring system of a responsible medical organization. A responsible medical organization is the parent organizational unit, such as an intensive care unit (ICU), emergency room (ER), or emergency medical sendee (EMS), that is responsible for providing the current medical care to and maintaining the medical records for the patient. Specifically, the system further includes communications port 215. One or more processors 210 further send the selection to a patient monitoring system of a responsible medical organization using communications port 215.

[0121] In various embodiments, the system further stores in a memory the time of the verbal command from the operator and the time of the patient selection in response. From the time difference between the verbal command and the patient selection, the system can make determinations about the patient's cognitive state or about the patient’s ability to use the system. The memory can be any type of memory accessible to one or more processors 210, including, but not limited to, local memory, memory that is part of a server or cloud system, or any type of memory described above.

[0122] In various embodiments, one or more processors 210 further send the time difference between the verbal command and the patient selection to a patient monitoring system of a responsible medical organization using communications port 215.

[0123] In various embodiments, the patient selection screen includes a yes area and no area. In various embodiments, the patient selection screen includes areas depicting different levels of pain, areas depicting different t pes of pain, areas depicting different bed controls, or areas depicting parts of a keyboard. In various embodiments, the patient selection screen includes areas depicting two or more graphics of a hand holding up different amounts of fingers of a hand for use as a Vanderbilt delirium test.

[0124] In various embodiments, the patient selection screen includes a graphic of the human body. When one or more processors 210 receive the selection of a body part from the human body graphic, one or more processors 210 display a zoomedin graphic of the body part on one or more patient displays 240 for further selection by the patient.Method for communicating with a nonverbal patient

[0125] Figure 15 is an exemplary flowchart showing a method 1500 for communicating with a nonverbal patient, in accordance with various embodiments.

[0126] In step 1510 of method 1500, an eye of patient is imaged using at least one eyetracker.

[0127] In step 1520, one or more patient selection screen options are displayed on one or more operator displays.

[0128] In step 1530, a patient selection screen option is received from one or more operator input devices.

[0129] In step 1540, a patient selection screen is displayed on one or more patient displays based on the received patient selection screen option.

[0130] In step 1550, the at least one eyetracker is instructed to determine a selection of the patient selection screen by the patient by monitoring a fixation of the eye of the patient on the patient selection screen.

[0131] In step 1560, the selection is received from the at least one eyetracker.

[0132] In step 1570, the selection is displayed on the one or more operator displays.System for monitoring the sleep of a patient

[0133] Figure 2 also depicts a system for monitoring the sleep of a patient, in accordance with various embodiments. The system includes at least one eyetracker 250 that images a closed eye of a patient, one or more patient displays 240, one or more operator displays 220, one or more operator input devices 230, and one or more processors 210.

[0134] One or more processors 210 receive a request to receive notification when the closed eye of the patient is opened from one or more operator input devices 230. One or more processors 210 instruct at least one eyetracker 250 to determine when the closed eye of the patient is opened. One or more processors 210 receive a notification from at least one eyetracker 250 that the closed eye of the patient has been opened. One or more processors 210 display a notification on one or more operator displays 220 that the closed eye of the patient has been opened.

[0135] In various embodiments, the system also monitors the medical staff operator in order to be able to establish communications as soon as the patient awakens.Specifically, one or more operator input devices 230 include camera 233 imaging the operator and microphone 234 that receives the operator’s voice, and at least one eyetracker 250 includes a speaker (not shown). One or more processors 210 receive a request to establish real-time communication with the patient from one or more operator input devices 230, and one or more processors 210 send realtime video of the operator to one or more patient displays 240 using camera 233 and send real-time audio of the operator using the speaker.

[0136] Note that, in Figure 2, one or more processors 210, one or more operator displays 220, and one or more operator input devices 230 are shown on the same mobile stand 260 as at least one eyetracker 250 and one or more patient displays 240. In various embodiments, one or more processors 210, one or more operator displays 220, and one or more operator input devices 230 can be in a separate room from at least one eyetracker 250 and one or more patient displays 240. As a result, establishing immediate real-time communications with the patient as soon as they awaken is very important.

[0137] In various embodiments, the system further includes communications port 215. One or more processors 210 further send a notification that the closed eye of the patient has been opened to a patient monitoring system of a responsible medical organization using communications port 215. In other words, the system automatically logs the moment the patient awakens by sending a notification to the patient monitoring system. As described above, this event can trigger other actions by the system, including, but not limited to, displaying information on one or more patient displays 240 meant to calm and or instruct the patient.

[0138] In various embodiments, the system further stores in a memory the time that the closed eye of the patient has been opened. In other words, the system records thefirst moment of consciousness. The memory can be any ty pe of memory accessible to one or more processors 210, as described above.

[0139] In various embodiments, one or more processors 210 further send the time that the closed eye of the patient has been opened to a patient monitoring system of a responsible medical organization using communications port 215.

[0140] In various embodiments, after receiving a notification from at least one eyetracker 250 that the closed eye of the patient has been opened, the operator can determine if the patient is nonverbal. If the patient is nonverbal, the system of Figure 2 can then be used for communicating with the nonverbal patient as described above.

[0141] In various embodiments, the system can further be used to determine when a patient closes their eyes for sleep or due to a loss of consciousness. Specifically, after receiving a notification from at least one eyetracker 250 that the closed eye of the patient has been opened, at least one eyetracker 250 can monitor the eye to determine if it closes for a predetermined period of time that indicates sleep or a loss of consciousness.

[0142] In various embodiments, one or more processors 210 further store in a memory and send the time that the patient goes to sleep or has lost consciousness to a patient monitoring system of a responsible medical organization using communications port 215.Method for monitoring the sleep of a patient

[0143] Figure 16 is an exemplary flowchart showing a method 1600 for monitoring the sleep of a patient, in accordance with various embodiments.

[0144] In step 1610 of method 1600, a closed eye of a patient is imaged using at least one eyetracker.

[0145] In step 1620, a request is received to receive notification when the closed eye of the patient is opened from one or more operator input devices.

[0146] In step 1630, the at least one eyetracker is instructed to determine when the closed eye of the patient is opened.

[0147] In step 1640, a notification is received from the at least one eyetracker that the closed eye of the patient has been opened.

[0148] In step 1650, a notification is displayed on one or more operator displays that the closed eye of the patient has been opened.

[0149] While the present teachings are described in conjunction with various embodiments, it is not intended that the present teachings be limited to such embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those of skill in the art.

[0150] Further, in describing various embodiments, the specification may have presented a method and / or process as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. As one of ordinary skill in the art would appreciate, other sequences of steps may be possible. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, the claims directed to the method and / or process should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the various embodiments.

Claims

WHAT IS CLAIMED IS:

1. A system for communicating with a nonverbal patient, comprising: at least one eyetracker that images an eye of a patient; one or more patient displays; one or more operator displays; one or more operator input devices; and one or more processors that display one or more patient selection screen options on the one or more operator displays, receive a patient selection screen option from the one or more operator input devices, display a patient selection screen on the one or more patient displays based on the received patient selection screen option, instruct the at least one eyetracker to determine a selection of the patient selection screen by the patient by monitoring a fixaton of the eye of the patient on the patient selection screen, receive the selection from the at least one eyetracker, and display the selection on the one or more operator displays.

2. The system of claim 1. wherein the one or more input devices comprise an operator microphone and wherein the one or more processors further monitor the operator microphone for a verbal command before instructing the at least one eyetracker to determine a selection of the patient selection screen by the patient.

3. The system of claim 1, further comprising a communications port, wherein the one or more processors further send the selection to a patient monitoring system of a responsible medical organization using the communications port.

4. The system of claim 3, wherein the responsible medical organization comprises an intensive care unit (ICU), an emergency room (ER), or an emergency medical service (EMS).

5. The system of claim 1, wherein the patient selection screen comprises ayes area and no area.

6. The system of claim 1, wherein the patient selection screen comprises areas depicting different levels of pain.

7. The system of claim 1, wherein the patient selection screen comprises areas depicting different types of pain.

8. The system of claim 1. wherein the patient selection screen comprises areas depicting different bed controls.

9. The system of claim 1, wherein the patient selection screen comprises areas depicting two or more graphics of a hand holding up different amounts of fingers of a hand for use as a Vanderbilt delirium test.

10. The system of claim 1, wherein the patient selection screen comprises areas depicting parts of a keyboard.

11. The system of claim 1, wherein the patient selection screen comprises a graphic of the human body.

12. The system of claim 10, wherein when the one or more processors receive the selection of a body part from the human body graphic, the one or more processors display a zoomed-in graphic of the body part on the one or more patient displays for further selection by the patient.

13. A method for communicating with a nonverbal patient, comprising: imaging an eye of patient using at least one eyetracker; displaying one or more patient selection screen options on one or more operator displays; receiving a patient selection screen option from one or more operator input devices; displaying a patient selection screen on one or more patient displays based on the received patient selection screen option; instructing the at least one eyetracker to determine a selection of the patient selection screen by the patient by monitoring a fixaton of the eye of the patient on the patient selection screen; receiving the selection from the at least one eyetracker; and displaying the selection on the one or more operator displays.

14. The method of claim 13, wherein the one or more input devices comprise an operator microphone and further comprising monitoring the operator microphone for a verbal command before instructing the at least one eyetracker to determine a selection of the patient selection screen by the patient.

15. The method of claim 13, further comprising sending the selection to a patient monitoring system of a responsible medical organization using a communications port.

16. The method of claim 15, wherein the responsible medical organization comprises an intensive care unit (ICU), an emergency room (ER), or an emergency medical service (EMS).

17. A system for monitoring the sleep of a patient, comprising: at least one eyetracker that images a closed eye of a patient; one or more patient displays; one or more operator displays; one or more operator input devices; and one or more processors that receive a request to receive notification when the closed eye of the patient is opened from the one or more operator input devices, instruct the at least one eyetracker to determine when the closed eye of the patient is opened, receive a notification from the at least one eyetracker that the closed eye of the patient has been opened, and display a notification on the one or more operator displays that the closed eye of the patient has been opened.

18. The system of claim 17, wherein the one or more operator input devices comprise a camera imaging the operator and a microphone that receives the operator’s voice and the at least one eyetracker comprises a speaker, wherein the one or more processors receive a request to establish real-time communication with the patient from the one or more operator input devices, and wherein the one or more processors send real-time video of the operator to the one or more patient displays using the camera and send real-time audio of the operator using the speaker.

19. The system of claim 17, further comprising a communications port, wherein the one or more processors further send a notification that the closed eye of the patient has beenopened to a patient monitoring system of a responsible medical organization using the communications port.

20. The system of 19, wherein the responsible medical organization comprises an intensive care unit (ICU), an emergency room (ER), or an emergency medical service (EMS).

21. The system of claim 17, wherein, after receiving a notification from the at least one eyetracker that the closed eye of the patient has been opened, the one or more processors further display one or more patient selection screen options on the one or more operator displays, receive a patient selection screen option from the one or more operator input devices, display a patient selection screen on the one or more patient displays based on the received patient selection screen option, instruct the at least one eyetracker to determine a selection of the patient selection screen by the patient by monitoring a fixaton of the eye of the patient on the patient selection screen, receive the selection from the at least one eyetracker, and display the selection on the one or more operator displays.

22. A method for monitoring the sleep of a patient, comprising: imaging a closed eye of a patient using at least one eyetracker; receiving a request to receive notification when the closed eye of the patient is opened from one or more operator input devices, instructing the at least one eyetracker to determine when the closed eye of the patient is opened;receiving a notification from the at least one eyetracker that the closed eye of the patient has been opened; and displaying a notification on one or more operator displays that the closed eye of the patient has been opened.

23. The method of claim 22, wherein the one or more operator input devices comprise a camera imaging the operator and a microphone that hears the operators voice and the at least one eyetracker comprises a speaker, and further comprising receiving a request to establish real-time communication with the patient from the one or more operator input devices and sending real-time video of the operator to the one or more patient displays using the camera and sending real-time audio of the operator using the speaker.

24. The method of claim 22, further comprising sending a notification that the closed eye of the patient has been opened to a patient monitoring system of a responsible medical organization using a communications port.

25. The method of claim 24, wherein the responsible medical organization comprises an intensive care unit (ICU), an emergency room (ER), or an emergency medical service (EMS).

26. The method of claim 22, wherein, after receiving a notification from the at least one eyetracker that the closed eye of the patient has been opened, further comprising displaying one or more patient selection screen options on the one or more operator displays, receiving a patient selection screen option from the one or more operator input devices, displaying a patient selection screen on the one or more patient displays based on the received patient selection screen option,instructing the at least one eyetracker to determine a selection of the patient selection screen by the patient by monitoring a fixaton of the eye of the patient on the patient selection screen, receiving the selection from the at least one eyetracker, and displaying the selection on the one or more operator displays.

Images