System and method for patient monitoring

Abstract

The present disclosure provides systems and methods for monitoring a patient's cognitive status based on ocular image data. The patient monitoring system includes a camera device configured to record images of the patient's eye, and a data processing subsystem in data communication with the camera and operable to (i) receive and process ocular image data from the camera, (ii) classify the ocular image data into gestures, determine gestures indicative of the patient's cognitive status, and (iii) transmit a signal to a remote device indicative of the cognitive status. The system may further include an actuator module and an output device, and the output may be an automated questionnaire.

Description

Technical Field

[0001] The present disclosure relates to a system that enables the evaluation of a patient's cognitive state based on the patient's eye image data.

Background Art

[0002] Lack of communication is a common unmet need for inpatients, especially in intensive care units. The lack of effective communication is thought to be a factor in the development of intensive care unit (ICU) delirium. Effective communication can be part of the prevention and treatment of a patient's ICU delirium. The current standard of communication for critically ill ventilated patients is limited to, for example, nodding, writing, and pointing to a communication board.

[0003] Systems and methods are known that enable a user's communication by tracking the user's eyes.

[0004] WO2016142933 discloses a system with a selection interface that selectively presents a user with a series of communication options. An optical sensor detects light reflected from the user's eyes, provides a correlation signal, and processes this signal to determine the relative eye orientation with respect to the user's head. Based on the determined relative eye orientation, the selected communication option is determined and implemented.

[0005] WO2019111257, which is incorporated herein by reference in its entirety, discloses a control system for communicating with an individual by tracking the eyes and / or by tracking other physiological signals generated by the individual. This system is configured to classify captured eye images into gestures similar to computer joystick-like operations. These gestures enable the user to operate a computer or system, for example, with menu items.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] Wearable devices can generally function as monitoring devices, particularly as emergency calling devices, to improve patient orientation. In addition, wearable devices can achieve more extensive and effective communication compared to the current standard communication for ICU patients.

Means for Solving the Problems

[0008] The system of the present disclosure helps to determine or monitor the cognitive state of patients who have temporarily or permanently lost the ability to communicate verbally and can be in various cognitive states. The patient population to which the system of the present disclosure applies are patients whose corresponding motor function is impaired. This impairment can be temporary, for example, in the case of patients in the ICU or patients recovering from trauma (such as an accident), or can be the result of a permanent pathological condition such as paralysis caused by a central or peripheral nervous system disease (such as ALS).

[0009] The present disclosure relates to a system comprising a camera that captures a series of image data of one or both of a patient's eyes. This eye image data is received and processed by a predetermined data processing subsystem, which functions to classify the eye image data and interpret it as eye gestures such as opening the eyes or blinking. The data processing subsystem then, based on a predetermined criterion (the criterion may be a fixed criterion or a dynamic criterion, for example, a criterion that varies across medical fields or geographical locations, or a criterion derived from medical books), determines specific gestures, gesture patterns, and a series of gestures among the interpreted eye gestures that characterize a state of interest such as arousal or delirium. Based on the determination, the data processing subsystem can determine that the patient is, is in, or has a likelihood of being in, or has a likelihood of coming into, a cognitive state. The subsystem that determines the state of interest reports the determination content to a remote device by transmitting a signal. For example, if it is indicated by opening the eyes that a patient admitted to the ICU has woken up for the first time in several days, the system transmits an alert signal to any one of or a combination of the connected nurse devices, medical staff members, or family members. The system disclosed in this document may be implemented in any medical or non-medical institution, including but not limited to rehabilitation facilities, nursing facilities, long-term emergency treatment facilities, and housing for the elderly.

[0010] In some embodiments, the system comprises a wearable headset, and the camera is equipped in a head unit configured to be attached to the patient's head.

[0011] The system can further include an actuator module configured to drive an output device to present an output to a patient, so the system can also provide the patient with some output including content, questionnaires, and feedback selected by the patient. For example, the system enables an automatic computer determination of the delirium state of an intensive care patient. Specifically, the system provides the patient with a computer interface for the established Intensive Care Unit (ICU) Confusion Assessment Method (CAM-ICU) test, and the patient can provide responses through the system (i.e., by the communication signals provided by the system). To facilitate the determination of the cognitive state, the system may be configured to receive and process additional physiological data other than the eye image data. In some embodiments, the system is also configured to receive and process additional data, such as verbal input from the user.

[0012] In some embodiments, the system can have high accuracy and short inference time to realize a simpler and more reliable communication system for people with impaired communication capabilities. This can be linked, among other things, to shortening the training session and more efficient operation of a Graphics Processing Unit (GPU), or, in some embodiments, to speeding up the data transmission to a remote processor (server), whereby the remote processor processes and analyzes the transmitted data more efficiently.

[0013] According to a first aspect of that aspect, a patient monitoring system for determining a patient's cognitive state is provided, the system comprising a camera configured to record an image of the patient's eye, and a data communication with the camera to (i) receive and process eye image data from the camera, (ii) classify the eye image data into gestures, determine a gesture indicating the patient's cognitive state, and (iii) transmit a signal communicating the cognitive state to a remote device. The system is developed and designed to monitor patients. This may include any in-patient, such as, for example, intensive care unit patients, ALS patients, locked-in syndrome patients, ventilator-dependent patients, critically ill patients, and patients lacking the ability to communicate verbally, etc. Further examples are non-in-patients under medical management or under the management of a caregiver.

[0014] The term monitoring includes continuous or intermittent monitoring. According to some embodiments, the monitoring is for a period of several seconds or minutes for the purpose of medical evaluation. According to other embodiments, the monitoring is long-term monitoring, for example, for patients admitted for several days, weeks, months, and years.

[0015] The term determination (or its derivatives) refers to any binary determination of the cognitive state (awake or non-awake, in pain or not in pain, delirious or not delirious, etc.), quantitative determination (e.g., duration of the waking state, total daily waking period, confusion or pain level on a numerical scale of 1 to 10, or likelihood index), or qualitative determination (relative quality of sleep, e.g., state of disorientation compared to a reference state yesterday, etc.). The determination can be based on the timing, order, duration, pattern, or other metrics of the patient's eye gestures. The determination includes the likelihood, onset, and duration in any metric (minutes, hours, or days) of the cognitive state. The determination may be derived from the gesture based on established criteria. The system may also have the option of input by a physician or other caregiver to determine the definition or determination rules.

[0016] According to some embodiments, the determination is a predictive determination including the likelihood that the patient will manifest a cognitive state.

[0017] In some embodiments, the determination is made based on a database collected based on the eye gestures of the patient himself or a group of patients. The database can be based on past or real-time eye gesture data.

[0018] The term cognitive state includes cognitive abnormalities such as wakefulness, sleep, delirium, decline, confusion, disorientation, abnormal attention or perception, memory impairment, distress, abnormal decision-making, dissatisfaction, discomfort, pain, and depression. The cognitive state may be the natural cognitive state of the patient, or may be a cognitive state induced, affected, or adjusted by medical interventions such as drugs.

[0019] In some embodiments, the patient monitoring system is independent of the screen.

[0020] In some embodiments, the head unit is a lightweight head-mounted device attached to the patient's head by a family member, caregiver, or the patient himself, and may further include a bone conduction speaker / headphone. The head unit can be easily removed. The camera may be equipped on the head unit and is configured to record eye image data including an image of one or both eyes, any eyelid, or both eyelids of the patient and generate image data representing the same.

[0021] In some embodiments, the camera is equipped on a head unit configured to be attached to the patient's head.

[0022] In some embodiments, the camera may be attached to a frame near the user, such as a bed frame, a frame holding medical equipment, among others.

[0023] In some embodiments, the camera is fixed to the patient's eye.

[0024] In some embodiments, the camera is not fixed to the patient's eye.

[0025] According to some embodiments, the camera is an infrared camera or a visible light camera.

[0026] Generally, the operation of the system of the present disclosure is not affected by lighting conditions.

[0027] According to some embodiments, the position of the camera (e.g., attached to the head unit) is fixed with respect to the patient's eye and serves as the only reference point for the captured image data.

[0028] According to some embodiments, communication based on the patient's eye is not to detect the exact location or position of what the patient is looking at or the corneal reflection (e.g., with respect to the screen), but is similar to a joystick-like operation. Also, according to the present disclosure, there is usually no need for a calibration procedure using the screen before use, and in fact, there is no need to use the screen for communication using the system at all.

[0029] The term "joystick-like operation" as described in this document refers to gesture classification including tracking the position of the pupil area in the eye image data.

[0030] According to some embodiments, the joystick-like operation follows the description of WO2019111257 which is incorporated herein by reference in its entirety.

[0031] According to some embodiments, the tracking of the position of the pupil area is performed independently of the patient's face.

[0032] According to some embodiments, the tracking of the position of the pupil area is performed based on a fixed camera.

[0033] According to some embodiments, the tracking of the position of the pupil area is performed based on a non-fixed camera.

[0034] The pupil area in the context of the present disclosure is either the pupil or any part of the pupil determined to indicate the pupil.

[0035] In some embodiments, the position of the pupil area is determined based on a database containing image data labeled with the pupil or eye gesture. The image data may be obtained from the patient himself or from any other patient or group of patients. In some embodiments, the position of the pupil area based on the labeled database is determined using a machine learning method, for example, a model that takes into account the likelihood that a given image data matches a specific gesture.

[0036] In some embodiments, the position of the pupil area may be determined based on its position within a limit map, and a specific position is determined when the pupil area touches the boundary of the limit map or the tangent of the boundary. For example, when the pupil area touches the upper boundary of the limit map, the image data is classified as an "up" gesture, or when the pupil area does not touch any boundary of the limit map, the image data is classified as a "straight" gesture. The limit map may be derived from a position map that includes areas within the movable range of the pupil area. In one example, the position map is set as a rectangle defined by the upper, lower, left, and right positions of the pupil area. In some embodiments, the limit map covers at least one area limited by a boundary that is at least 20%, 40%, 60%, 80%, 90%, 95% away from the center of the position map. The limit map is usually at least 80% away from the center of the position map. The position map may be obtained based on the patient's image data or on any database containing image data with or without labeled gestures. Optionally, the position map is within a larger region of interest (ROI) defined based on anatomical features of the eye or its surroundings.

[0037] Sometimes, the position of the pupil area may be determined based on its position within a limit map that includes two or more key zones. The key zones may be small, separate, arbitrarily selectable, overlapping areas within the eye image data. Sometimes, when the pupil area is arranged to touch the boundary of at least one key zone, touch the tangent of the boundary, or be included within at least one key zone, the pupil position is classified as a gesture. The image may be captured at a frame rate of at least 30 Hz using a camera with a shutter speed of at least 1 / 30 sec. In some embodiments, when the pupil maintains its gesture determination position for at least a predetermined period, e.g., for 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1, or 2 seconds, the pupil position is classified as a gesture.

[0038] In some embodiments, the position of the pupil area is determined using computer vision tools, arbitrarily based on circular, curved, and straight features. Optionally, the pupil area determination is performed based on determining dark or black pixels within the image data. In some embodiments, the center of the pupil is calculated to be used as a criterion when classifying eye gestures. In some embodiments, optionally, the movement range of the eye relative to the calculated center of the pupil is also calculated to be used as a criterion for the "central" eye gesture.

[0039] In some embodiments, the position map is a discontinuous map, or a map that includes two or more areas or one zone of the eye image.

[0040] In some embodiments, eye gestures are classified based on the image data of a single image data frame. In other embodiments, eye gestures are classified based on multiple image data frames, and as an optionally selectable matter, are classified based on 2, 5, 10, 20, or 50 image data frames. In other embodiments, gestures are classified based on 60%, 70%, 80%, 85%, 90%, 95%, or 97% of multiple image data frames within a predetermined time frame or among a predetermined number of image data frames.

[0041] In some embodiments, the position of the pupil area is determined using an object detection tool. Optionally, a deep learning model is combined with object detection to determine eyes, pupils, and their features. Optionally, the eye features to be determined are selected from the iris of the eye, the outer (edge) circle or iris, the pupil, the inner pupil area, the outer pupil area, the upper eyelid, the lower eyelid, the corner of the eye, or any combination thereof.

[0042] Optionally, the image data is classified into eye gestures based on any one or combination of eye features. The eye features may be selected from the iris of the eye, the pupil, the inner pupil area, the outer pupil area, the outer (edge) circle or iris, the upper eyelid, the lower eyelid, and the corner of the eye.

[0043] In some embodiments, the eye determination or eye features are derived from bounding box object detection. The term bounding box may relate to an area defined by two longitudes and two latitudes. Optionally, the latitude is a decimal number in the range of -90 to 90, and the longitude is a decimal number in the range of -180 to 180. In some embodiments, the detection is performed based on labeled data including bounding box coordinates and image data labels.

[0044] In some embodiments, the position of the pupil area is determined by using a machine learning tool or by using a combination of a computer vision tool and a machine learning tool.

[0045] Optionally, the combined computer vision and machine learning model is based on a Single Shot Detector (SSD) algorithm.

[0046] Optionally, the combined computer vision and machine learning model is based on a You Only Look Once (YOLO) algorithm. The YOLO algorithm may include two steps. The first step includes using an object detection algorithm to detect at least two, three, four, or five key zones in the eye image data. The second step includes using a supervised machine learning algorithm to determine the pupil of the eye in the eye image data based on the coordinates of the key zones. In some embodiments, the position of the pupil area is determined based on a machine learning model. Optionally, the model is based on the image data of a patient. Optionally, the model is based on the image data of a patient population. Optionally, the model is based on the image data of a healthy population.

[0047] Optionally, the model is a supervised, semi-supervised, or unsupervised model.

[0048] In some embodiments, the supervised model is based on image data that is manually labeled or automatically labeled.

[0049] In some embodiments, the boundaries between different gestures are defined based on the image data of the patient.

[0050] In some embodiments, gesture classification is based on using machine learning methods. Specifically, the machine learning model may be a neural network model consisting of a number of linear transformation layers followed by element-wise non-linearity. Classification may include the evaluation of eye characteristics for an individual patient or across multiple patients. In some embodiments, classification estimates the range of eye movement. The machine learning model can use a classifier selected from logistic regression, support vector machine (SVM), or random forest.

[0051] In some embodiments, the model is a deep learning model, and optionally, a convolutional neural network (CNN) model. In some embodiments, the model classifies image data into at least five basic gestures. Optionally, the basic gestures are selected from blinking, up, down, left, right, center (straight), upper left diagonal, upper right diagonal, lower left diagonal, and lower right diagonal.

[0052] In some embodiments, the gesture classification has an accuracy of at least 80%, 85%, 90%, 95%, 97%, 98, or 99%.

[0053] In some embodiments, the average inference time of the gesture classification is up to 100ms, 125ms, 150ms, 175ms, 200ms, 240ms, 250ms, 260ms, 270ms, 300ms, 350ms, or any value in between. In some embodiments, the average inference time of the gesture classification is in the range of 100 - 200ms, for example 125 ± 25ms. The inference time corresponds to 3 - 10 frames per second (fps), and sometimes 8fps.

[0054] In some embodiments, gestures are classified based on the majority of gestures within a predetermined time frame (e.g., in the range of 50 - 200 ms). A basic time frame is set, and gestures are classified within several consecutive time frames, and a gesture is defined when the same classification occurs in the majority of those time frames. For example, when a 125 - ms frame is applied and the same gesture is classified in 2 out of 3 images, the gesture is determined every 375 ms (i.e., after 3 such frames). This has the advantage of avoiding overly sensitive responses and enabling a more stable output.

[0055] In some embodiments, as an optionally selectable matter, the machine - learning model is further used for headset placement by classifying whether at least one eye is present in the image frame. As an optionally selectable matter, the image data is classified into three classes: at least one eye is fully within the frame, there is no eye within the frame, and at least one eye is in the center of the frame.

[0056] In some embodiments, first, the machine - learning model is used for headset placement by identifying the position of at least one eye in the image data. Next, gesture classification is performed using a combination of computer - vision tools and machine - learning tools.

[0057] In some embodiments, the data - processing subsystem is a distributed or non - distributed parallel subsystem.

[0058] In some embodiments, the classification of the image data into gestures and the determination of gestures indicating the patient's cognitive state are performed by components of a distributed subsystem.

[0059] In some embodiments, the data processing subsystem communicates with the camera and optionally wirelessly, receives and processes image data from the camera, and further classifies the image data into gestures (optionally, the classification is performed according to the above-described joystick-like operation). The gestures may include voluntary or involuntary eye gestures. The gestures may include straight, central, right, left, up, down, upper left diagonal, upper right diagonal, lower left diagonal, and lower right diagonal positions of the pupil, a series of pupil positions, closing the eyes, opening the eyes, curvilinear eye movements, eye movements behind closed eyelids (e.g., rapid eye movements during sleep), an increase or decrease in pupil size (e.g., dilated or constricted pupils), an enlargement or constriction of a part or the interior of the pupil, eyelid spasms, blinking, and a series of eyelid blinks. The gestures may also include any eye or eyelid movement related to eyelid disorders such as ptosis, eyelid retraction, a decrease or increase in blinking, and apraxia of eyelid opening.

[0060] Optionally, the gestures may relate to either one or both eyes.

[0061] Optionally, the gestures may include a series of two or more eyelid blinks.

[0062] The gestures may be selected from any one or combination of eye gestures well-known in the art. For example, the gestures may be fixations (static gestures or gazes) or a series of fixations and their durations, gestures or points of fixation, and their clusters and distributions.

[0063] In some embodiments, the system requests the patient to perform a straight gesture between other gestures.

[0064] In some embodiments, the blink gesture is determined by using a machine learning model that classifies the image data of the eye as an image of a closed eye as a region of dark pixels or, optionally (by supervised or unsupervised learning), an artificial intelligence model.

[0065] In some embodiments, the opening gesture is determined by using a machine learning model that classifies the eye image data of an open or not-closed eye as an image of the pupil area or, optionally, as an artificial intelligence model or an eye image that can be arbitrarily selected.

[0066] In some embodiments, the closing gesture is determined as a series of closed-eye image data frames following at least one image of a not-closed eye.

[0067] In some embodiments, the rapid eye movement (REM) gesture (common in REM sleep) is determined as a series of image data frames with rapid changes between the image data frames. As a non-limiting example, a series of gestures "down - center - down - center - down - center" will be the image data classified as the REM gesture.

[0068] In some embodiments, when the pupil area touches the boundary of the limit map, or the tangent of the boundary, or is included within the limit map, the gesture is classified. The image may be captured at a frame rate of at least 30 Hz using a camera with a shutter speed of at least 1 / 30 sec. In some embodiments, when the pupil maintains its gesture determination position for at least a predetermined period, for example, for 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1, or 2 seconds, the pupil position is classified as a gesture.

[0069] In some embodiments, any one of the gesture type, the number of gestures, the duration of the gesture, and the corresponding signal and output is determined by the patient or caregiver.

[0070] In some embodiments, the signal is transmitted based on a single gesture, multiple gestures, or a sequence of gestures. Optionally, the signal is transmitted based on the timing, order, duration, pattern, other metrics of the patient's eye gestures, and any combination thereof.

[0071] In some embodiments, an eye-opening gesture of at least 1, 3, 5, 7, 10, 30, 60 seconds initiates an "awakening" signal.

[0072] In some embodiments, an eye-closing gesture of at least 1, 3, 5, 7, 10, 30, or 60 seconds initiates a "sleep" signal.

[0073] In some embodiments, a series of 1, 2, 3, or 5 blinks can initiate a "request for assistance" signal.

[0074] In some embodiments, a series of up to 10 blinks within 30 seconds selects a signal.

[0075] In some embodiments, an opening or closing of at least one eye for 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or 30 seconds transmits a signal. Optionally, the signal is a sleep mode signal to the system.

[0076] In some embodiments, the signal reports to the caregiver the patient's waking period over 24 hours.

[0077] In some embodiments, the signal is a recommendation provided by the system regarding the timing of physically visiting the patient's room.

[0078] According to another non-limiting embodiment, a patient with limited capabilities can use one gesture, optionally based on their own definition or range of eye movement, for example, using only a "left" gesture, to operate the system.

[0079] In some embodiments, the gesture that enables the patient to operate the computer is to look in a predetermined direction in the normal way (eye gesture) instead of the gesture of the patient looking at a specific location (eye fixation). For example, even when the patient is not concentrating the field of view on a specific physical or virtual object, a general left gaze can serve as a gesture.

[0080] In some embodiments, the determination of the gesture indicating the patient's cognitive state is made based on rules derived from general medical knowledge and methods well-known in the art, or based on criteria set by a nurse, doctor, caregiver, or patient. The rules may be preset or may be derived after the initial monitoring of an individual patient. The rules implemented for a patient may be derived from the patient himself or from a typical population group. In some embodiments, the determination rules are one or more fixed rules (based on comprehensive medical knowledge) or dynamic rules (e.g., based on a single edition of a medical book or different rules across medical fields and geographical regions, etc.). In some embodiments, the rules are dynamic in the sense that they are derived based on an online artificial intelligence algorithm. The determination may be made as part of monitoring the patient's condition (at a specific time point or continuously) or for predicting the cognitive state. The prediction determination includes the likelihood that the patient will explicitly exhibit a cognitive state.

[0081] The cognitive state may be determined in combination with some physiological or non - physiological data. In some embodiments, the cognitive state is determined based on any combination of eye image data received by the system and physiological data. In some embodiments, the cognitive state is determined based on any combination of eye image data received by the system, physiological data, and further data. The further data may be any quantitative, qualitative, probabilistic, text, individual, or group data. The data may be received online or at a predetermined time point. The data may be from the patient, a medical staff member, or a family member.

[0082] Finally, when the system determines the cognitive state, the data processing subsystem transmits a signal related to the cognitive state to the remote device.

[0083] In some embodiments, the signal may be any signal, and optionally, it may be a digitized computer signal (e.g., to set the system to a sleep mode or to operate a mono Internet of Things (IOT) device), or optionally, it may be a visual signal (a change in lighting from green to red or vice versa), an auditory signal (an alarm sound, etc.), or other digital signals such as text by a text messaging service.

[0084] In some embodiments, the signal is, optionally, a word, symbol, or sentence that is pre-determined by a patient or caregiver or configured by the patient, using a menu that optionally selects an interface by characters.

[0085] In some embodiments, the remote device is an alert device in an intensive care unit, a device for nurses, or a device carried by a caregiver.

[0086] In some embodiments, the signal to the remote device is transmitted by wireless communication. The term wireless communication as used in this document may include any form of communication independent of a conductor connection, such as Wi-Fi communication, a mobile communication system, Bluetooth communication, infrared communication, and radio wave communication. In some embodiments, the wireless communication is via a wireless network such as a wireless personal area network (WPAN) or a wireless body area network (WBAN).

[0087] As a non-limiting example, a signal to a remote device may correspond to triggering a device or equipment in the patient's room, such as turning on or dimming the indoor lighting, starting light therapy, playing media content such as sound or music, or controlling the room temperature.

[0088] As a non-limiting example, a signal to a remote device may be a signal that activates a system for improving or alleviating a patient's medical condition, emotional state, or cognitive state (e.g., a system for reducing delirium).

[0089] In some embodiments, image data of opening at least one eye is classified as an eye-opening gesture indicating a waking state, and an alert signal is transmitted to the nurse device.

[0090] In some embodiments, the patient monitoring system further includes an actuator module configured to drive an output device to present an output to the patient, and the data processing subsystem records reaction image data representing reaction eye movements following the output, classifies the reaction image data into reaction gestures, and determines a gesture indicating the patient's cognitive state.

[0091] The term "output" includes any sensory output or content for the patient.

[0092] In some embodiments, the output is provided through an Internet of Things (IoT) device such as a smart home device.

[0093] In some embodiments, the output is related to the prevention or reduction of ICU delirium as an optionally selectable matter of the cognitive state.

[0094] In some embodiments, the output is related to inducing or promoting a cognitive state such as a sleep state.

[0095] (From recording of image data to transmission of cognitive state signal) In addition to the above-described automatic operation, the output may be started in response to an eye gesture or a signal. Additionally, the output may be selected based on a cognitive state automatically determined by the system, whereby, as an arbitrarily selectable matter, it acts positively on the patient's cognitive state. For example, in response to the determination of an anxious cognitive state, relaxing music, the voice of a family member, white noise, or cognitive practice may be selected.

[0096] In some embodiments, the sensory output is visual (e.g., a message or question on a screen), auditory (such as voice guidance or a question), tactile output (such as a contact stimulus to the patient's foot), or any combination thereof.

[0097] In some embodiments, the term "content" is any content, including general content or content created personally. These may include general medical information videos or messages distributed to the patient by the patient's doctor, nurse, or caregiver.

[0098] In some embodiments, the content is some media content selected by the patient.

[0099] In some embodiments, the media content is selected to improve the patient's functional performance and medical adaptation, for example, to reduce stress.

[0100] In some embodiments, the media content is some visual or auditory content familiar to the patient, such as the voices of family members or a known environment. The auditory content may be a pre-recorded media file or may be transmitted online. According to one embodiment, the content is a menu system that allows the patient to move through a menu, which is presented to the patient and controlled by the patient's eye gestures by selecting menu items using eye gestures. The presentation of the menu may be an audible presentation (by means of a loudspeaker, earphone, headset, embedded audible device, etc.) or a visual presentation (by means of a display on a screen, a small display in front of the patient, etc.). The menu may be hierarchical, which means that the selection of a menu item may open other lower-level selectable options.

[0101] In some embodiments, the menu follows the description of WO2019111257, which is hereby incorporated by reference in its entirety.

[0102] In some embodiments, the output is a human, digital, or automated questionnaire. Sometimes, the type and content of the questionnaire are determined based on the patient's response to a previous question. Sometimes, the type of the questionnaire and the questions of the questionnaire are determined based on response gestures. According to one embodiment, the questionnaire is the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). According to one embodiment, a gesture of opening at least one eye starts the output of the CAM-ICU.

[0103] In some embodiments, the output is a video or audio that shows the patient the date, time, and location of the patient.

[0104] In some embodiments, the output instructs the user on how to respond using eye gestures (as an optionally selectable matter with respect to the questionnaire).

[0105] The term reaction image data refers to image data recorded in response to the output provided to the patient.

[0106] The term "responsive eye movement" refers to eye movements recorded in response to output provided to a patient.

[0107] The term "responsive eye gesture" refers to an eye gesture classified based on responsive eye movement or responsive image data. In some embodiments, the responsive gesture indicates the cognitive state of a patient.

[0108] In one embodiment, a gesture of opening at least one eye initiates the CAM-ICU output, and a responsive eye gesture indicates a delirious state. As a non-limiting example, the system conducts the CAM-ICU test by outputting a multiple-choice question, and the patient conveys their answer by blinking.

[0109] As another non-limiting example, the system conducts a reorientation assessment by outputting the current time, date, and location, and the patient responds by maintaining an eye-opening gesture for a predetermined period.

[0110] Also, as another exemplary and non-limiting embodiment, a patient can be prompted to select from several options by auditory or visual output, for example, as one selection "up" (i.e., an upward gesture), another selection "down", etc. According to a further exemplary and non-limiting embodiment, options can be presented to the patient (e.g., by auditory output), thereby prompting the patient to gesture in a specific or unspecified direction, blink a series of times, close the eyelids for a predetermined period, etc. when a particular option is presented. The latter is useful, for example, for the rapid selection of characters for writing text. This embodiment can also serve patients responding to a questionnaire.

[0111] According to another exemplary and non - limiting embodiment, the questionnaire is a pain scale arbitrarily selected from a numerical evaluation scale, the Stanford pain scale, the Brief Pain Inventory, the Wong - Baker Faces, the Global Pain Scale, the Visual Analogue Scale, and the McGill Pain Index. A further questionnaire according to the present disclosure is an air hunger or dyspnea questionnaire.

[0112] In some embodiments, the data processing subsystem further functions to receive and classify one or more physiological parameters and determine the physiological parameters indicating the patient's cognitive state, or any combination of the gesture and physiological parameters.

[0113] In some embodiments, the determination includes the likelihood that the patient will manifest a cognitive state based on a combination of the eye gesture and physiological parameters. In some embodiments, the combination is rapid eye movement accompanied by a high heart rate.

[0114] In other embodiments, the cognitive state is determined based on a series of gestures accumulated over a certain period on a scale of minutes, hours, days, or months.

[0115] The scale of minutes includes up to 10, 20, 30, 60, 120, or 240 minutes.

[0116] The scale of days includes up to 1, 2, 3, 4, 5, 7, 14, 30, or 60 days.

[0117] The scale of months includes 1, 2, 4, 6, 8, 12, or 24 months.

[0118] Sometimes, the cognitive state is evaluated based on involuntary or continuous gestures made by the patient, independent of the input provided by the system.

[0119] The term "physiological parameter" includes any sample of physiological measurements that can be obtained from a patient's body, including any signals obtained from the patient's nerves, heart, somatosensory, vocal, and respiratory systems, as well as the movement of selected muscles. Physiological parameters may be recorded by any sensor equipment or measurement device, microphone, spirometer, galvanic skin response (GSR) device, touch or pressure probe, skin conductance probe, electroencephalography (EEG) device, electrocorticogram (ECoG) device, electromyography (EMG), electrooculography (EOG), and electrocardiogram.

[0120] In some embodiments, the physiological parameter, or any combination of the eye gesture and the physiological parameter, indicates the cognitive state of the patient.

[0121] In some embodiments, a patient monitoring system is provided that includes a plurality of patient monitoring systems (or subsystems). In some embodiments, each of the plurality of patient monitoring systems monitors a separate patient.

[0122] In some embodiments, the system further includes a centralized processor that receives signals representing the cognitive state from each of the patient monitoring systems (or subsystems) and classifies such signals according to one or more predetermined criteria. In some embodiments, the classification performed by the centralized processor is based on criteria set by a physician or caregiver. The criteria may represent considerations regarding medical urgency, time, space, and any combination thereof.

[0123] According to one non-limiting example, a group of patients admitted to the same medical department is being monitored by the plurality of patient monitoring systems (alternatively, subsystem, each patient is being monitored by a subsystem), the signals transmitted by the system are classified by a centralized processor, and the nurse device is receiving alerts ranked based on medical urgency.

[0124] In some embodiments, the signal is an integrated signal reporting the wakefulness state of at least 2, 4, 6, 8, 10, 20, 50, 100 patients.

[0125] In some embodiments, the signal is an integrated signal reporting the sleep state of at least 2, 4, 6, 8, 10, 20, 50, 100 patients.

[0126] According to some embodiments of the present disclosure, the system has an optionally selectable matter measured as a time of 5, 10, 20, 30, 60, 120, and 360 minutes up to which a patient needs to use the device to execute at least one signal, from the first time the system is set up with respect to the patient, for example from proper headset wearing, and is accompanied by a training period.

[0127] In some embodiments, the data processing subsystem further serves to receive and classify some additional (physiological or non-physiological) data. A non-limiting example is to receive and classify a patient's response (e.g., by a reactive eye gesture or otherwise) to a questionnaire (which can be optionally selected, and the questionnaire is output by the system) in order to obtain the cumulative result of the questionnaire. As an example, a patient can respond to a questionnaire regarding stress level through the system, and the results of the questionnaire are received and classified by the system and, optionally, combined with any one or a combination of eye image data, physiological data, and additional data. The additional data may be any one or a combination of non-medical data such as population data (e.g., epidemiological data), individual data (genetic predisposition), medical history data, or socioeconomic characteristics. The additional data may be sent directly to the data processing subsystem or, as a non-limiting example, may be sent via a device for nurses. In some embodiments, the data processing subsystem further serves to receive and classify additional data and determine some additional data indicating the patient's cognitive state, or a combination of the gesture and additional data.

[0128] In some embodiments, the additional data, or any combination of the eye gesture, physiological parameters, and additional data, indicates the patient's cognitive state.

[0129] In some embodiments, the data processing subsystem receives and processes auditory data (e.g., by natural language processing). For example, when a patient is questioned by another person, e.g., a caregiver, the data processing subsystem receives and processes the speech of the doctor and, based on the speech context analysis of the other person and the patient's language, can propose a response to the patient in the patient's own language, including guidance for the patient. This embodiment enables a patient in a foreign country to easily communicate with local doctors and caregivers.

[0130] According to a second aspect of that aspect, a method for determining a patient's cognitive state is provided, the method comprising: (a) recording image data of at least one of the patient's eyes; (b) classifying the image data into gestures; (c) determining a gesture indicating the patient's cognitive state; and (d) transmitting a signal conveying the cognitive state to a remote device.

[0131] In some embodiments, the method further comprises verifying an output to the patient.

[0132] In some embodiments, the method further comprises: (a) recording reaction image data representing reaction eye movements following the output; (b) classifying the reaction image data into reaction gestures; and (c) determining a gesture indicating the patient's cognitive state.

[0133] In some embodiments, the method further comprises: (a) receiving and classifying one or more physiological parameters; and (b) determining the gesture and physiological parameters indicating the patient's cognitive state, or any combination thereof.

[0134] In some embodiments, a method for determining a patient's cognitive state is provided, the method comprising: (a) recording a patient's eye image from a plurality of patient monitoring systems; (b) classifying the image data into gestures; (c) determining a gesture indicating the patient's cognitive state to obtain a determined cognitive state; and (d) classifying the determined cognitive state according to one or more predetermined criteria.

[0135] According to some embodiments, a method for integrated patient monitoring for determining the cognitive state of a plurality of patients is provided. The method includes: (a) recording eye images of each patient from a plurality of patient monitoring systems; (b) classifying the image data from each of the systems into gestures; (c) determining a gesture indicative of the cognitive state of each patient to obtain a determined cognitive state; (d) classifying the determined cognitive state according to one or more predetermined criteria; and (e) transmitting an integrated signal conveying the cognitive state to a remote device.

[0136] To better understand the subject matter disclosed herein and to illustrate how it may be actually implemented, embodiments will be described solely by way of non-limiting examples with reference to the accompanying drawings.

Brief Description of the Drawings

[0137]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0138] First, refer to FIG. 1, which shows a schematic block diagram of a patient monitoring system according to a non-limiting example of the present disclosure. The patient monitoring system 100 includes a camera 104 provided in a head unit 102 configured to be attached to a patient's head. The camera may be provided in some fixed frame near the patient. The camera 104 functions to continuously capture images of one or both of the patient's eyes and eyelids and generate image data representing the same. The system 100 includes a parallel or distributed data processing subsystem 106 that communicates with the camera 104 via data. The data processing subsystem 106 receives and processes eye image data from the camera, classifies the eye image data into gestures, and determines a gesture indicating the patient's cognitive state. Next, the data processing subsystem 106 transmits a signal conveying the cognitive state to a remote device.

[0139] As a non-limiting example, an ICU patient who has been unconscious and hospitalized for several days is being monitored by the patient monitoring system 100. The caregiver places the wearable head unit 102 on the patient's head. When the patient first opens their eyes, the movement of the patient's eyes is captured by the camera 104. The image data from the camera 104 is received by the data processing subsystem 106. Next, the image data is classified into gestures, and if an eye-opening gesture is classified, an awake state is indicated and a "help requested" signal is wirelessly transmitted to the nearest nurse's device.

[0140] FIG. 2 shows a block diagram of the system of the present disclosure, which further includes an actuator module 108 that drives a first output device 110. The output device 110 may be a visual display device, for example, a digital screen, or may be an audible device, for example, a speaker, headphones, etc.

[0141] As a non-limiting example, a hospitalized patient suspected of delirium is wearing the head unit 102 and is being monitored by the patient monitoring system 100. When the patient blinks twice, the eye movement of the patient is photographed by the camera 104 and classified as a series of two-blink gestures by the data processing subsystem 106. Based on the gesture, the output of the digital CAM-ICU questionnaire by the output device 110 driven by the actuator module 108 starts. The patient responds to the CAM-ICU assessment by performing reactive eye movements, and the reactive eye movements of the patient are photographed by the camera 104 and classified as reactive eye gestures by the data processing subsystem 106. These reactive eye gestures are interpreted as indicating whether the patient is actually in a delirious state. When the delirious state is actually determined by the data processing subsystem 106, a delirium signal is sent by the data processing subsystem 106 to the patient's doctor.

[0142] Figures 3 and 4 show non-limiting and exemplary components of the system of the present disclosure. Numerical indices that are shifted by 100 from the indices of the components shown in FIG. 1 are assigned to the components of FIGS. 3 and 4. For example, the head unit shown as 102 in FIG. 1 is shown as 202 in FIG. 3. Therefore, the reader can refer to the above text for details regarding the functions of these components.

[0143] FIG. 3 shows a non-limiting example of a system including a head unit 202, a camera 204, and a decentralized data processing subsystem 206 that wirelessly communicates (e.g., Wi-Fi, Bluetooth) with a remote device.

[0144] FIG. 4 shows a non-limiting illustration of a system that can be worn by a prospective patient.

[0145] FIG. 5 shows a schematic system architecture according to a non-limiting, exemplary embodiment. According to this exemplary system, the system includes a data processing subsystem 306 and a headset 302. The system communicates bidirectionally remotely with (i) a medical staff server 308 that communicates with a remote medical staff station 310 (e.g., via an IoT protocol), (ii) a device settings cloud server 312 via Wi-Fi communication, and (iii) a web-based additional application 314 via Bluetooth communication. The medical staff server includes a system database 316, an event scheduler 318 (e.g., enabling medical staff to schedule calendar events for a specific patient, such as a "good morning" greeting at 08:00 every day, or the start of a CAM-ICU or other questionnaire every 12 hours), a server 320 that stores and retrieves data (media files for voice menus, World Wide Web (WWW) pages 322, etc.), and a text-to-speech application programming interface (API) 324. The staff server receives data, e.g., voice messages, via a remote family portal 326. The family portal 326 can send recommendations to a device settings cloud server with a web portal 328. The device settings cloud server includes voice banking 330 (generating original content using synthesized voice based on recorded voice), text-to-speech 332, and translation 334 APIs, as well as a user database 336.

[0146] The system comprises an output device and an actuator module that drives an output selected from a set consisting of questionnaires, audible questions and answers, an orientation message (location, date, time), music, family records, etc. The output may be triggered by eye gestures classified by the system. In addition, the response to the output may be provided by the patient with a reaction gesture classified by the data processing subsystem. The response may include answering a question. Overall, the system realizes a more natural, gentle and well - managed environment for the patient, thereby improving the quality of hospitalization and reducing negative emotions during hospitalization such as a feeling of lack of control, anxiety, fear of inability to communicate, etc. The system is also linked to a secure remote support cloud 338 to realize a reverse tunnel to the services of technicians located remotely.

[0147] Figure 6 shows an example of a possible screen display (dashboard) at a medical staff station that displays the following. - A communication log including the patient's answers to questions sent from the station in the form of (spoken) voice. - A communication module that presents communication options to the medical staff - The patient's sleep / wake pattern - A log of all alerts and alerts that require physical intervention by staff in the user's room (user assistance request alerts, camera position changed, device disconnected from the network, etc.) - A reminder of the device location within the department - A log of the activities of the user and the device - The results of the questionnaire (CAM - ICU, pain scale, etc.) - Buttons for music, orientation, and recordings of family voices

Claims

1. A patient monitoring system for determining a patient's cognitive state, the patient monitoring system comprising: a camera device configured to record an image of the patient's eye; communicating data with the camera device, i. receiving and processing eye image data from the camera device, ii. classifying the eye image data into gestures and determining a gesture indicating the patient's cognitive state for predicting the patient's cognitive state, iii. transmitting a signal indicating the cognitive state to a remote device, a functioning data processing subsystem; an actuator module; and an output device, wherein the actuator module is configured to drive the output device to present an output to the patient, the output including media content triggered by an eye gesture classified by the patient monitoring system and including any visual or auditory content familiar to the patient, the media content being selected based on the predicted cognitive state by the patient monitoring system, thereby acting positively on the patient's cognitive state, the output further including one or more questions, or an automated questionnaire, and instructions to the patient regarding a method of using an eye gesture to respond to the one or more questions, or the automated questionnaire, the data processing subsystem recording reaction image data representing reaction eye movements in response to the output, classifying the reaction image data into reaction gestures, and determining a gesture indicating the patient's cognitive state to determine the patient's current cognitive state, which is a delirious state, in response to the output. A patient monitoring system.

2. The system according to claim 1, wherein the camera device is equipped in a head unit configured to be attached to the patient's head.

3. The patient monitoring according to any one of claims 1 to 2, wherein the remote device is an alert device in an intensive care unit, a device for a nurse, or a device carried by a caregiver.

4. The patient monitoring system according to any one of claims 1 to 3, wherein the gesture is selected from at least one eye opening, at least one eye closing, pupil position, a series of pupil positions, and a series of eyelid blinks.

5. The patient monitoring system according to any one of claims 1 to 4, wherein an eye-opening gesture of at least one eye indicates a waking state and an alert signal is transmitted to the nurse device.

6. The patient monitoring system according to any one of claims 1 to 5, wherein the signal to the remote device is transmitted by wireless communication.

7. The patient monitoring system according to any one of claims 1 to 6, wherein the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) is started by the opening of at least one eye.

8. The data processing subsystem further receives and classifies one or more physiological parameters, and functions to determine the physiological parameter indicating the cognitive state of the patient, or any combination of the gesture and the physiological parameter. The patient monitoring system according to any one of claims 1 to 7.

9. A patient monitoring system comprising a plurality of patient monitoring systems according to any one of claims 1 to 8.

10. The system according to claim 9, further comprising a centralized processor that receives signals representing the cognitive state from each of the patient monitoring systems and classifies the signals according to one or more predetermined criteria.

Images