System and method for predicting likelihood of falling or degree of anesthesia recovery

Abstract

To predict the likelihood of falling or the degree of anesthesia recovery.SOLUTION: A system includes: at least one camera installed at a predetermined location in a hospital to capture an image; a motion detector configured to detect the motion of a patient in the image; a face recognizer configured to recognize the face of the patient in the image to determine the identity of the patient, and recognize the expression and gaze of the patient; and a monitor configured to predict the likelihood of falling or the degree of anesthesia recovery by using an action of the patient detected by the motion detector, the identity of the patient determined by the face recognizer, and the expression and gaze of the patient.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system and method for predicting the likelihood of a fall or the extent of anesthesia recovery, and more particularly to a system and method for predicting the likelihood of a fall or the extent of anesthesia recovery using a video-based recognizer and artificial intelligence. [Background technology]

[0002] In recent years, hospitals, especially general hospitals, have been getting larger. Larger hospitals have the advantage of being able to provide more diverse and higher quality services to patients, but the disadvantage is that it is difficult to closely monitor patients who fall in beds, hallways, or recover from anesthesia due to the large number of patients. In particular, high-risk patients need to be closely monitored because they are at high risk of falling or recovering from anesthesia, but there are limitations due to issues such as a shortage of manpower.

[0003] Currently, medical staff are deployed directly to the scene to monitor patients, but in places where it is difficult to constantly deploy or constantly manage medical staff, such as corridors, restrooms, and patient rooms, it is difficult to respond in real time to accidents such as patient falls and accidents during anesthesia recovery. In addition, measures to increase response capabilities by adding more medical staff or increasing their working hours may increase costs and rapidly increase the fatigue level of medical staff, resulting in reduced efficiency.

[0004] In this regard, Korean Patent Publication No. 10-2013-0118512 (hereinafter, Patent Document 1) discloses a patient condition monitoring system using face recognition and a method for providing a patient condition monitoring service using the same. Patent Document 1 discloses a configuration in which a surveillance camera is provided to obtain a patient's facial image, the patient's facial image and vital sign information are received, whether or not an emergency situation has occurred for the patient is detected, and a message is generated based on the result and transmitted to the terminals of the patient's medical staff and caregivers. However, Patent Document 1 requires additional equipment to detect an emergency situation using vital signs, and therefore cannot be applied to all patients and does not fundamentally solve the problem.

[0005] Therefore, there is currently a demand for a system and method that can be applied using only a camera, installs a video-based recognizer, and automatically identifies patients and detects fall movements or the degree of anesthesia recovery, and can detect multiple patients simultaneously, and predicts the possibility of a fall or the degree of anesthesia recovery. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2013-0118512 Summary of the Invention [Problem to be solved by the invention]

[0007] One object of the present invention is to provide a system and method that uses a video-based recognizer to automatically identify a patient and detect fall movements or the degree of anesthesia recovery, and predict the possibility of a fall or the degree of anesthesia recovery.

[0008] In addition, the present invention predicts the possibility of a fall or the degree of anesthesia recovery using only images, and is intended to be applicable to any location where imaging equipment is installed.

[0009] The present invention also seeks to sense the likelihood of falls or the degree of anesthesia recovery for multiple patients simultaneously. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the present invention is characterized in that it includes one or more cameras that are installed at predetermined positions within a hospital to capture images; a motion detection unit that detects the movement of a patient in the image; a face recognition unit that recognizes the face of the patient in the image to identify the patient and recognizes the patient's facial expression and gaze; and a monitoring unit that predicts the possibility of a fall or the degree of anesthesia recovery using the patient's behavior detected by the motion detection unit, as well as the patient's identity identified by the face recognition unit, and the patient's facial expression and gaze.

[0011] Preferably, the monitoring unit may include a facial expression recognition module that recognizes facial expressions and facial expression intensities that are correlated with situations in which a fall may occur or with the degree of anesthesia recovery.

[0012] Preferably, the monitoring unit may include an eye gaze recognition module that classifies the recognized patient's eye gaze into one or more of saccades, vergence movements, smooth pursuit movements, and vestibulo-ocular movements based on the movements.

[0013] Preferably, the gaze recognition module is capable of calculating the difference between the segmented patient gaze and normal gaze and detecting abnormal gaze movements.

[0014] Preferably, the monitoring unit can detect an area in which a patient is present from the image captured by the camera, and detect the patient's posture by extracting the patient's skeleton within the area.

[0015] Preferably, the behavior recognition module can recognize the meaning of each behavior stage from the detected change in posture of the patient.

[0016] Preferably, the behavior recognition module can detect abnormal behavior by calculating a difference between a detected posture of the patient and a normal posture according to the meaning of each behavior stage.

[0017] Preferably, the monitoring unit can predict the possibility of a fall using an artificial neural network trained on the patient's gaze direction and skeletal movement features extracted from footage of a possible fall or footage of a fall occurring.

[0018] Preferably, the monitoring unit classifies whether the identified patient is in a high-risk group for the occurrence of a fall or recovery from anesthesia, and if the patient is in the high-risk group, makes a prediction with increased sensitivity.

[0019] Another feature of the present invention is that it includes the steps of capturing video using one or more cameras installed at predetermined positions in the hospital, detecting the patient's movements in the video, recognizing the patient's face in the video to identify the patient and recognizing the patient's facial expression and gaze, and predicting the possibility of a fall or the degree of anesthesia recovery using the patient's behavior detected in the step of detecting the patient's movements, and the patient's identity identified in the step of recognizing the patient's facial expression and gaze, and the patient's facial expression and gaze. Effect of the Invention

[0020] The present invention has the advantage of being able to automatically detect the possibility of a fall or the degree of anesthesia recovery in real time even in an environment where there is no or a shortage of medical staff.

[0021] The present invention has an advantage in that it can simultaneously detect the possibility of a fall or the degree of anesthesia recovery for all patients photographed by the camera. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram of a system for predicting the possibility of a fall or the degree of anesthesia recovery in accordance with an embodiment of the present invention. [Diagram 2]FIG. 2 is a configuration diagram of a monitoring unit according to an embodiment of the present invention. [Diagram 3] 13 is a specific flowchart for predicting the possibility of a fall or the degree of anesthesia recovery by a monitoring unit according to an embodiment of the present invention. [Figure 4] 11A to 11C are diagrams for explaining how a monitoring unit predicts the possibility of a fall by comprehensively taking into account the gaze direction and skeletal movement in accordance with an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an input / output structure of an artificial neural network used in a monitoring unit according to an embodiment of the present invention. [Figure 6] 1 is a flowchart of a method for predicting the likelihood of a fall or the extent of anesthesia recovery according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The present invention will now be described in detail with reference to the accompanying drawings, in which: FIG. 1 is a block diagram of a semiconductor device according to an embodiment of the present invention; FIG. 2 is a block diagram of a semiconductor device according to an embodiment of the present invention;

[0024] The objects and effects of the present invention will be understood or become more apparent from the following description, and the objects and effects of the present invention are not limited to the following description. In addition, when describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may obscure the gist of the present invention, the detailed description will be omitted.

[0025] The terms used in the present invention are merely for describing specific embodiments and are not intended to limit the present invention. The singular term includes the plural term unless otherwise clearly indicated in the context. In this application, the terms "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the description of the invention, and should be understood not to preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] Terms such as first, second, etc. may be used to describe various components, but these components should not be limited by the terms. These terms are used only to distinguish one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component, without departing from the scope of the present invention.

[0027] Unless otherwise specified, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in the present invention.

[0028] When interpreting elements, even if there is no specific explicit description, it is interpreted as including the margin of error. In the explanation of the time relationship, for example, when the time sequence is explained using "after", "following", "next to", "before", etc., it includes cases where it is not consecutive, unless "directly" or "indirectly" is used.

[0029] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the accompanying drawings.

[0030] 1 is a block diagram of a system 100 for predicting the possibility of a fall or the degree of anesthesia recovery according to an embodiment of the present invention. Referring to FIG. 1, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery may include one or more cameras 110, a motion detection unit 120, a face recognition unit 130, and a monitoring unit 140.

[0031] System 100, which predicts the possibility of a fall or the degree of anesthesia recovery, can automatically detect the possibility of a fall or the degree of anesthesia recovery in real time, even in environments where there is no medical staff or a shortage of medical staff, and can respond to patient falls and other accidents in real time even in places where constant management is difficult, thereby reducing fatigue among medical staff.

[0032] The system 100 for predicting the possibility of a fall or the degree of anesthesia recovery has the advantage of being able to simultaneously detect the possibility of a fall or the degree of anesthesia recovery for all patients photographed by the camera.

[0033] The system 100 for predicting the possibility of a fall or the degree of anesthesia recovery is installed in a place where it is difficult to constantly deploy medical staff, and can automatically identify patients and predict the possibility of a fall or the degree of anesthesia recovery in real time. The system 100 for predicting the possibility of a fall or the degree of anesthesia recovery can identify patients and detect the possibility or occurrence of a fall and send it to medical staff, even in a space where the situation cannot be known unless medical staff actually enters, such as a toilet in a hospital room, as long as only a camera is installed. In addition, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery can optimally respond to patients at high risk of falling by using specific patient linkage information (disease, medication status, etc.) in a specific hospital room from EMR data.

[0034] The system 100 for predicting the possibility of a fall or the degree of recovery from anesthesia can simultaneously recognize the movement / face of each of the multiple patients in the video, and therefore can predict the possibility of a fall or the degree of recovery from anesthesia for multiple patients in the video. In one embodiment, when multiple patients are monitored in a place where multiple patients and visitors are mixed, such as a hospital corridor, it is possible to detect the fall of a specific patient from the video by simultaneously performing movement / face recognition. In this case, the system 100 for predicting the possibility of a fall or the degree of recovery from anesthesia can further lower the minimum limit of fall movement recognition if the patient is determined to be in a high risk group for falling (patients in a radiation therapy ward, patients administered medicine that causes frequent urination, patients with fractures, etc.) through linkage with patient information after face recognition, so that the system can warn of falls in advance even if the patient's movement is small.

[0035] As another embodiment, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery can detect the occurrence of a patient fall event from a bed at a specific position in the video by automatic motion / face recognition when there are multiple beds and multiple patients are monitored, such as in a hospital room or an emergency center. Furthermore, as another embodiment, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery can detect the occurrence of a patient fall event from a bed at a specific position in the video by only automatic motion recognition when there are multiple beds and multiple patients are monitored, such as in a hospital room or an emergency center. Furthermore, as another embodiment, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery can be used in a case where there are multiple beds and the anesthesia state of multiple patients is monitored, such as in an endoscopy room at a medical examination center. By using automatic facial recognition for the patients in each bed (or specific bed-patient linked information), the system can receive linked information for a specific patient (anesthesia-related examination history, examination items that affect anesthesia, etc.) from EMR data, confirm the different degrees of anesthesia recovery for each patient, automatically detect movements for each specific patient during the anesthesia recovery stage, and provide information regarding each of the movements to medical staff when movements occur within the normal recovery time range or when no movements occur.

[0036] The system 100 for predicting the possibility of a fall or the degree of anesthesia recovery may include an image sensor such as a camera. Although RFID (Radio Frequency IDentification) and UWB (Ultra-wideband) sensors can identify a person, they cannot identify multiple people individually when they move at the same time, and cannot identify individual body movements, so each person must carry an RFID or UWB sensing device. Radar can identify body movements, but cannot identify a person individually. In contrast, the system 100 for predicting the possibility of a fall or the degree of anesthesia recovery according to an embodiment of the present invention uses an image sensor to individually identify the identities and body movements of multiple people at the same time, so each person does not need to carry a sensing device.

[0037] The camera 110 is installed at a predetermined position in the hospital and can capture images. One or more cameras 110 may be installed at predetermined positions in the hospital. The camera 110 can capture images in real time and provide the captured images to the motion detection unit 120 or the face recognition unit 130.

[0038] One or more cameras 110 may be placed at corners, intersections, hallways, patient rooms, and other locations where there is a risk of a patient falling, or in a post-anesthesia care center.

[0039] The motion detector 120 can detect the movement of the patient in the video. The motion detector 120 can recognize the patient's behavior, such as the patient's posture and changes in posture, through automatic motion recognition.

[0040] The motion detector 120 can recognize the patient's actions and detect the patient's movements in real time. Specifically, the motion detector 120 can detect the patient's movements using artificial intelligence.

[0041] The face recognition unit 130 can recognize the face of a patient in the video and identify the patient. The face recognition unit 130 can identify the patient's identity by comparing the face of the patient recognized from the video with patient information in a database. The face recognition unit 130 can recognize the faces of not only the patient in the video but all people in the video and identify them. For example, the face recognition unit 130 can identify the identities of patients, general visitors, medical staff, general employees, etc.

[0042] The face recognition unit 130 can recognize the face of a patient in real time to identify the patient. Specifically, the face recognition unit 130 can detect the patient's movement using artificial intelligence. The face recognition unit 130 can provide a high accuracy (99.9% or more) that is robust against time and environmental changes. The face recognition unit 130 can provide a fast and simple identity authentication function through high-speed / high-efficiency face recognition. The face recognition unit 130 provides a strong anti-spoofing function and can distinguish whether a face being captured is a real person or a fake image. The motion detection unit 120 and the face recognition unit 130 can utilize existing video devices such as CCTV installed in the hospital as they are. The identity of the patient identified by the face recognition unit 130 may be used to determine whether the identified patient is at high risk of falling or anesthesia recovery.

[0043] The face recognition unit 130 can recognize the facial expression and gaze of the patient. Here, the facial expression of the patient means a facial appearance showing an emotion such as a grimace, pain, confusion, etc., and the gaze of the patient may mean a direction in which the eyes are looking.

[0044] The artificial intelligence for motion sensing, face recognition, and monitoring mentioned may be a DNN (Deep Neural Network) or CNN (Convolutional Neural Network) based model. A convolutional neural network (CNN) is a type of deep learning model for processing data having a lattice pattern such as an image, focusing on the structure of the animal's visual cortex. A convolutional neural network may generally include a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer and the pooling layer may exist repeatedly in the neural network, and the input data may be transformed as output through these layers. The convolutional layer uses a kernel (or mask) for feature extraction, and the element-wise product of each element of the kernel and the input value is calculated at each position and summed to obtain an output value. This is called a feature map. Such a procedure may be repeated while applying multiple kernels to form any number of feature maps. In a convolutional neural network, the convolutional and pooling layers perform feature extraction, while the fully connected layer maps the extracted features to a final output, such as a classification operation.

[0045] Neural networks such as convolutional neural networks may be trained to minimize output errors. In addition to the forward propagation process of extracting values ​​from an input layer to an output layer, backpropagation occurs in a neural network to calculate the error between input training data and the corresponding output value of the neural network, and to update the weights for the nodes of each layer to reduce such errors. The training process of a convolutional neural network may be summarized as a process of searching for a kernel that extracts an output value with the least error based on given training data. The kernel is the only parameter that is automatically learned in the training process of the convolutional layer. Meanwhile, in a convolutional neural network, the kernel size, the number of kernels, padding, etc. are hyperparameters that must be set before starting the training process, and therefore, different convolutional neural network models may be classified according to the kernel size, the number of kernels, the number of convolutional layers, and the number of pooling layers.

[0046] 2 is a block diagram of the monitoring unit 140 according to an embodiment of the present invention. Referring to FIG. 2, the monitoring unit 140 may include a facial expression recognition module 141, a gaze recognition module 142, and a behavior recognition module 143.

[0047] The monitoring unit 140 can predict the possibility of a fall or the degree of anesthesia recovery by using the patient's behavior detected by the motion detection unit 120, the patient's identity, and the patient's facial expression and gaze determined by the face recognition unit 130. The monitoring unit 140 can predict the possibility of a fall or the degree of anesthesia recovery by analyzing face recognition such as identity determination, facial expression recognition, and gaze recognition, and movement recognition such as posture, changes in posture, movement speed, and stopping time. Unlike conventional methods, the monitoring unit 140 can predict the possibility of a fall or the degree of anesthesia recovery by risk level anytime and anywhere using only the captured image.

[0048] FIG. 3 is a specific flowchart for predicting the possibility of a fall or the degree of anesthesia recovery by the monitoring unit 140 according to an embodiment of the present invention. Referring to FIG. 3, the monitoring unit 140 can detect the patient's movement from the image being captured in real time, identify the patient's identity by face recognition, and recognize the facial expression and gaze. The monitoring unit 140 detects an area in the image where the patient is present, detects the patient's skeleton within the area, identifies the posture, and recognizes the meaning of each action stage from the change in posture. Then, the monitoring unit 140 can detect the patient's movement by calculating the difference between a normal posture having a meaning for each action stage and the posture of the patient.

[0049] The monitoring unit 140 can recognize facial expressions associated with the possibility of falling or the degree of anesthesia recovery, and the intensity of the facial expressions (the higher the intensity, the greater the association). The monitoring unit 140 can also classify the gaze into functions according to movement, and recognize the patient's gaze by calculating the difference between the classified gaze and a normal gaze. Here, the patient's identity may be used only for the purpose of determining the high-risk group.

[0050] The monitoring unit 140 can finally predict the possibility of a fall or the degree of anesthesia recovery by combining behavior recognition, facial expression recognition, and gaze recognition. The monitoring unit 140 can predict the possibility of a fall and detect the occurrence of a fall in order to prevent the fall.

[0051] When predicting the possibility of a fall to prevent it, the monitoring unit 140 can use an artificial neural network model that has been trained in advance with typical gaze directions, facial expressions, and human skeletal movement actions that may cause a fall. For example, before taking a specific action, a person looks at the position to which they are going to move or the object they are going to grab, and at the same time takes the action. When the monitoring unit 140 detects typical gaze directions, facial expressions, and the action itself that may cause a fall, it can determine that the patient is at risk of falling, and when performing a specific action, it can recognize that there is a problem with the synchronization of gaze and movement (deficiency of vestibular oculomotor movement) and determine that the patient is at risk of falling.

[0052] FIG. 4 is a diagram for explaining that the monitoring unit 140 according to the embodiment of the present invention predicts the possibility of a fall by comprehensively considering the gaze direction and skeletal movement. Referring to FIG. 4, in one embodiment of predicting the possibility of a fall to prevent a fall, the act of getting out of bed without lowering the bed rail is accompanied by the action of standing with the gaze facing the rail to grab the rail. Therefore, the monitoring unit 140 can predict the risk of a fall by having an artificial neural network learn such a risk in advance or by creating a database. In addition, the monitoring unit 140 can detect the risk of a fall in advance by recognizing that the direction of the gaze to grab the rail is not continuously maintained toward the rail due to the movement of the head or body due to a defect in vestibular eye movement (a movement of the gaze to stand in the opposite direction to the movement of the head so that the gaze can be directed in a certain direction to balance when the head moves).

[0053] In another embodiment, the act of reaching out from inside the bed to grab an object outside the bed involves the movement of reaching out the arm while directing the gaze toward the object to grab it. Therefore, the monitoring unit 140 can predict the risk of falling by having the artificial neural network learn such risks in advance or by storing them in a database.

[0054] Furthermore, in another embodiment, the monitoring unit 140 can predict the risk of falling by having the artificial neural network learn about the following risks in advance or by creating a database: when the bed rail is pushed down, but the caregiver's bed (which has wheels and is slippery) is stepped on while looking at it in order to get out of the bed; when the caregiver looks at the space around the bed where there is no railing and tries to get out of the narrow space; when the caregiver gets out of bed by himself and looks at the wheelchair (which has wheels and is slippery) while trying to sit on it; when the patient tries to get up from the toilet bowl without looking at the support bar or trying to grab it; when the patient walks in a ward corridor in a way that is different from the way a normal person walks; when the patient's caregiver (who can be identified by facial recognition as the caregiver) does not help the patient walk in the ward corridor by looking at his or her smartphone, for example.

[0055] In order to detect the occurrence of a fall, the monitoring unit 140 can use an artificial neural model that is pre-trained according to typical gaze directions (floor, ceiling) and human skeletal movement actions (fall, collapse, slip, etc.) that occur during or after a fall. The monitoring unit 140 can detect the occurrence of a fall and allow medical staff to take prompt follow-up measures.

[0056] The facial expression recognition module 141 can recognize facial expressions and the intensity of facial expressions that are associated with a situation in which a fall may occur or a degree of recovery from anesthesia. A specific facial expression of a patient is associated with a specific situation. The specific facial expressions of a patient include a frown, pain, fear, confusion, doubt, closing the eyes, a blank expression, and a blank expression. The facial expression recognition module 141 can recognize how much the current facial expression of a patient is associated with the possibility of a fall or the degree of recovery from anesthesia by using the relevance of the facial expression that occurs at that time to the situation in which a fall may occur or the degree of recovery from anesthesia that is stored in a database in advance. The facial expression recognition module 141 can output how much the current facial expression of a patient is associated with the possibility of a fall or the degree of recovery from anesthesia by using an artificial neural network that is trained by the characteristics of the patient's facial expression depending on the situation in which a fall may occur or the degree of recovery from anesthesia.

[0057] The gaze recognition module 142 can classify the recognized gaze of the patient into one or more of saccades, vergence movements, smooth pursuit movements, and vestibulo-ocular movements according to the movement. Among the classified gaze movements, vergence movements, smooth pursuit movements, and vestibulo-ocular movements, excluding saccades, are gaze functions that are highly related to a person's conscious behavior. Therefore, the gaze recognition module 142 can recognize three gaze movements that are highly related to a person's conscious behavior, and if abnormal, can be recognized as a precursor symptom of the possibility of a fall or a high risk of anesthesia recovery. In particular, vestibular-ocular movements are gaze movements that occur in the opposite direction to the head movement so that the gaze is directed in a certain direction in order to maintain balance when the head moves, and defects in vestibular-ocular movements are directly related to falls. The gaze recognition module 142 can segment gaze movements from the current patient's gaze using an artificial neural network trained on one or more features of any of the four gaze movements.

[0058] The gaze recognition module 142 can detect abnormal gaze movements by calculating the difference between the classified gaze of the patient and the normal gaze. The gaze recognition module 142 can detect abnormal gaze movements based on the degree of agreement between the normal gaze movements stored in the database and the current gaze movements of the patient in terms of direction, magnitude, etc. The gaze recognition module 142 can detect abnormal gaze movements using an artificial neural network trained by the characteristics of normal or abnormal gaze movements.

[0059] The behavior recognition module 143 can detect the area where the patient is located from the image captured by the camera, extract the skeleton of the patient in the area, and detect the posture of the patient. The behavior recognition module 143 can extract the skeleton of the patient in the image using an artificial intelligence model. Specifically, the artificial intelligence model for extracting the skeleton can extract the skeleton by extracting each joint of the human body and connecting the joints.

[0060] The behavior recognition module 143 can recognize the meaning of each behavior stage from the detected change in posture of the patient. The camera 110 may capture an image in real time, and the posture of the patient may be detected for each frame from the image captured in real time. Therefore, the behavior recognition module 143 can continuously compare the detected postures of the patient for each frame to detect a change in posture over time, and can recognize the meaning of each stage of the behavior the patient is about to take from the continuous change in posture. For example, the behavior recognition module 143 can recognize the meaning of each stage of the behavior from continuous changes in posture, such as an action of raising the upper body from a bed or a chair in the ward, an action of holding a handrail of a bed or an armrest of a chair, an action of putting the legs out of the bed, an action of lowering the legs to the floor of the hospital room while holding the floor of the bed or the handrail of the bed, an action of raising the body while holding the armrest of a chair, an action of standing beside the bed or in front of a chair, an action of walking toward an exit or a toilet, an action of walking down a corridor of a ward, etc.

[0061] That is, the behavior recognition module 143 can recognize the meaning of each behavior stage by comparing the meaning of each behavior stage according to the posture change stored in the database with the current posture change of the patient. The behavior recognition module 143 can detect the meaning of each behavior stage by using an artificial neural network trained based on the meaning of each behavior stage and the characteristics of the current posture change of the patient.

[0062] The behavior recognition module 143 can detect abnormal behavior by calculating the difference between the detected posture of the patient and a normal posture according to the meaning of each behavior stage. The behavior recognition module 143 can detect abnormal behavior based on the difference between the normal posture according to the meaning of each behavior stage stored in the database and the current posture of the patient. The behavior recognition module 143 can detect abnormal behavior using an artificial neural network trained by features of normal or abnormal behavior.

[0063] 5 shows an input / output structure of an artificial neural network used in the monitoring unit 140 according to an embodiment of the present invention. Referring to FIG. 5, the monitoring unit 140 can predict the possibility of a fall by using an artificial neural network trained with the gaze direction and skeletal movement features of a patient extracted from a video showing a possibility of a fall or a video showing a fall. In this case, data used to train the artificial neural network may include gaze direction features such as gaze features classified according to movement, differences from normal gaze, or associated facial expressions and facial expression intensity recognition, and skeletal movement features such as differences between meanings according to action stages and normal postures.

[0064] The monitoring unit 140 classifies whether the identified patient is in a high risk group for the occurrence of a fall or anesthesia recovery, and if the patient is in the high risk group, the monitoring unit 140 can increase the sensitivity for prediction. The monitoring unit 140 can use information on the high risk group for falling or the high risk group for anesthesia recovery, which are likely to suffer fatal injuries or fall, to predict the possibility of a fall or the degree of anesthesia recovery.

[0065] The monitoring unit 140 may notify medical staff and caregivers of the stage of the possibility of a fall based on the meaning of the behavior and the result of the behavior difference, rather than simply notifying the possibility of a fall as on / off. For example, the monitoring unit 140 may notify the possibility of a fall for each behavior stage, and may combine the result of the difference between the behavior stage and normal behavior to notify the possibility of a fall.

[0066] When the monitoring unit 140 detects the possibility of a fall, the occurrence of a fall, or the degree of anesthesia recovery, it can transmit a fall risk stage, a fall occurrence notification, or anesthesia recovery stage to medical staff. In addition, when transmitting the fall risk stage, fall occurrence notification, or anesthesia recovery stage to medical staff, the monitoring unit 140 also transmits linkage information (disease, medication status, etc.) of a specific patient in a specific hospital room from EMR data, enabling optimal treatment for the patient.

[0067] Fig. 6 is a flowchart of a method for predicting the possibility of a fall or the degree of anesthesia recovery according to another embodiment of the present invention. Referring to Fig. 6, the method for predicting the possibility of a fall or the degree of anesthesia recovery may include a step of capturing an image with one or more cameras installed at predetermined positions in a hospital, a step of detecting the movement of a patient in the image, a step of identifying the patient by recognizing the face of the patient in the image, and a step of predicting the possibility of a fall or the degree of anesthesia recovery using the behavior of the patient detected in the step of detecting the movement of the patient, and the identity of the patient, the facial expression and the gaze of the patient identified in the step of recognizing the facial expression and gaze of the patient.

[0068] The step of capturing images with one or more cameras installed at predetermined positions in the hospital refers to the operation performed by the one or more cameras 110 described above.

[0069] The step of detecting the patient's movement in the image refers to an operation performed by the movement detector 120 described above.

[0070] The steps of recognizing the patient's face in the image, identifying the patient, and recognizing the patient's facial expression and gaze refer to the operations performed by the face recognition unit 130 described above.

[0071] The step of predicting the possibility of a fall or the degree of anesthesia recovery using the patient's behavior detected in the step of detecting the patient's movements and the patient's identity, facial expression and gaze identified in the step of recognizing the patient's facial expression and gaze refers to an operation performed by the monitoring unit 140 described above.

[0072] Although the present invention has been described in detail above by taking representative embodiments, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by all modifications or alterations derived from the concept equivalent to the appended claims.

Claims

1. A system for predicting the likelihood of a fall or the extent of anesthesia recovery, comprising: one or more cameras installed at predetermined locations within the hospital to capture video; A motion detector that detects the movement of the patient in the image; A face recognition unit that recognizes the face of a patient in the video to identify the patient and recognize the patient's facial expression and gaze; A monitoring unit that predicts the possibility of a fall or the degree of anesthesia recovery using the patient's behavior detected by the motion detection unit, the patient's identity specified by the face recognition unit, and the patient's facial expression and gaze; Including, the system.

2. The monitoring unit is A facial expression recognition module that recognizes facial expressions and facial expression intensities that are correlated with possible fall occurrences or the degree of anesthesia recovery; The system of claim 1 , comprising:

3. The monitoring unit is a gaze recognition module for classifying the recognized gaze of the patient into one or more of saccades, vergence movements, smooth pursuit movements, and vestibular-ocular movements according to the movements; The system of claim 1 , comprising:

4. The gaze recognition module includes: Calculate the difference between the patient's gaze and normal gaze and detect abnormal gaze movements.

4. The system according to claim 3 .

5. The monitoring unit is a behavior recognition module that detects an area where a patient is present from the image captured by the camera, extracts a skeleton of the patient within the area, and detects a posture of the patient; The system of claim 1 , comprising:

6. The behavior recognition module includes: Recognizing the meaning of each behavioral stage from the detected changes in the patient's posture.

6. The system according to claim 5 .

7. The behavior recognition module includes: Calculating the difference between the detected posture of the patient and a normal posture according to the meaning of each behavior stage to detect abnormal behavior.

7. The system according to claim 6.

8. The monitoring unit is Predicting the possibility of a fall using an artificial neural network trained on the patient's gaze direction and skeletal movement features extracted from a video showing a possible fall or a video showing a fall; 2. The system of claim 1 .

9. The monitoring unit is Classify the identified patient as being at high risk for falls or anesthesia recovery; In the case of a high-risk group, the sensitivity is increased for prediction.

2. The system of claim 1 .

10. 1. A method for predicting the likelihood of a fall or the extent of anesthesia recovery, comprising: capturing video with one or more cameras installed at predetermined locations within the hospital; detecting movement of the patient in the video; identifying the patient by recognizing the patient's face in the image and recognizing the patient's facial expression and gaze; predicting the likelihood of a fall or the degree of anesthesia recovery using the patient's behavior detected in the step of detecting the patient's movement, and the patient's identity and the patient's facial expression and gaze identified in the step of recognizing the patient's facial expression and gaze; A method comprising:

Images