Patient Monitoring Systems

Abstract

This feature pertains to a patient monitoring system configured so as to be capable of appropriately monitoring the state of a patient. This patient monitoring system comprises: an estimation unit that estimates the state of the patient by inputting, into a first learning model, vitals information indicating vital signs of the patient and video analysis information obtained by analyzing video in which the patient appears; and a monitoring unit that monitors the state of the patient on the basis of the estimation results of the estimation unit. This feature can be applied to a monitoring system provided inside an ICU, for example.

Description

[Technical Field]

[0001] The present technology relates to a patient monitoring system, and more particularly to a patient monitoring system that enables appropriate monitoring of a patient's condition. [Background technology]

[0002] In medical settings, patients' conditions can change suddenly. This is especially true in ICUs, where many patients are in life-threatening or post-surgical conditions, making it necessary to develop methods for appropriately monitoring patients.

[0003] For example, Patent Document 1 describes a technique for monitoring a monitor screen displayed on a biological monitor and highlighting the time when an abnormality occurs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-233042 Summary of the Invention [Problem to be solved by the invention]

[0005] However, doctors and nurses do not understand a patient's condition by simply looking at changes in vital signs. In addition to changes in vital signs, doctors and nurses also sense signs of abnormalities or a sudden change in the patient's condition by observing the patient's external appearance and feeling something strange from experience.

[0006] The present technology has been developed in light of these circumstances, and makes it possible to appropriately monitor the condition of a patient. [Means for solving the problem]

[0007] A patient monitoring system according to one aspect of the present technology includes an estimation unit that estimates the condition of the patient by inputting vital information indicating the patient's vital signs and video analysis information obtained by analyzing video of the patient into a first learning model, and a monitoring unit that monitors the condition of the patient based on the estimation result by the estimation unit.

[0008] In one aspect of the present technology, vital information indicating a patient's vital signs and video analysis information obtained by analyzing video of the patient are input into a first learning model to estimate the patient's condition, and the patient's condition is monitored based on the estimation result. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing an embodiment of a patient monitoring system to which the present technology is applied; [Figure 2] FIG. 10 is a diagram illustrating an example of data acquired by an information processing device. [Figure 3] FIG. 10 is a diagram showing a flow of processing performed by an information processing device. [Figure 4] FIG. 10 is a diagram showing the flow of a method for extracting feature amounts around the eyes. [Figure 5] FIG. 10 is a diagram showing the flow of a method for extracting features of a face and shoulders. [Figure 6] FIG. 1 is a diagram showing an example of time-series data used in multivariate analysis. [Figure 7] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 8] FIG. 10 is a diagram showing an example of camera placement for each patient. [Figure 9] FIG. 10 is a diagram illustrating another example configuration of a patient monitoring system. [Figure 10] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 11] FIG. 10 is a diagram showing an example of a learning data set for a learning model used in each analysis. [Figure 12] 10 is a flowchart illustrating processing by the information processing device. [Figure 13]FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present technology will be described in the following order. 1. Patient Monitoring System 2. Configuration of information processing device 3. Operation of information processing device 4. Variations

[0011] <1. Patient monitoring system> FIG. 1 is a diagram showing an embodiment of a patient monitoring system to which the present technology is applied.

[0012] The patient monitoring system detects abnormalities and predicts sudden changes in a patient's condition by analyzing the appearance features obtained from video footage of the patient, in addition to the patient's vital sign data, diagnostic data, and attribute data. The results of abnormality detection and sudden change prediction are provided to medical professionals, including doctors and nurses.

[0013] The patient monitoring system of FIG. 1 is configured by connecting cameras 1A to 1C and medical devices 2A to 2C to an information processing device 3 via wired or wireless communication. A monitor 4 is also connected to the information processing device 3. The devices constituting the patient monitoring system are installed, for example, in an ICU (Intensive Care Unit). In the example of FIG. 1, three beds are installed in the ICU, each used by a patient A to C. The cameras 1A to 1C and medical devices 2A to 2C may be connected to the information processing device 3 via a network. The cameras 1A to 1C and medical devices 2A to 2C may also be connected to the information processing device 3 via an IP converter that converts the data into a predetermined communication protocol (for example, IP (Internet Protocol)). The IP converter includes an information processing circuit including a CPU and a memory.

[0014] Cameras 1A to 1C are configured, for example, as pan-tilt-zoom cameras that can change the imaging direction and angle of view. Cameras 1A to 1C capture images and acquire images of patients A to C, respectively. As images of patients A to C, for example, RGB images are acquired by cameras 1A to 1C. Note that hereinafter, when there is no need to distinguish between cameras 1A to 1C, they will simply be referred to as camera 1. The same applies to other devices that are provided in multiple numbers.

[0015] The medical devices 2A to 2C detect the vital signs of the patients A to C, respectively, and acquire time-series data of the vital signs for a predetermined period as vital sign data (vital information).

[0016] For example, one camera 1 and one medical device 2 are provided for each patient. In Fig. 1, three cameras 1 and three medical devices 2 are provided in the ICU, but in reality, the number of cameras 1 and medical devices 2 provided in the ICU corresponds to the number of patients and beds.

[0017] The information processing device 3 is a device that detects abnormalities and predicts sudden changes in the patient's condition and monitors the patient's condition. The information processing device 3 acquires various data from devices in the ICU, other systems, other systems in the hospital, etc.

[0018] FIG. 2 is a diagram showing an example of data acquired by the information processing device 3. As shown in FIG.

[0019] As shown in A of Figure 2, attribute data indicating the patient's gender, age, medical history, etc., and medical data indicating the results of blood tests, etc., are acquired from other systems within the ICU and other systems within the hospital. The medical data may be, for example, time-series data of the results of blood tests performed at predetermined intervals.

[0020] As shown in FIG. 2B, time-series data such as electrocardiogram, respiratory rate, SpO2, body temperature, blood glucose level, and invasive arterial pressure are acquired from medical device 2 as vital sign data.

[0021] As shown in C of Fig. 2, a facial image showing the patient's face and a whole-body image showing the patient's whole body are acquired from camera 1. Frame images constituting the video showing the patient are acquired as the facial image and the whole-body image.

[0022] The information processing device 3 in Figure 1 extracts appearance features of a patient from a face image and a whole-body image acquired by the camera 1. The information processing device 3 estimates the patient's condition by performing analysis based on the appearance features, attribute data, medical data, and vital sign data. The information processing device 3 also records the appearance features, medical data, and vital sign data.

[0023] The monitor 4 displays a list of images, vital sign data, estimated results of the patient's condition, etc. for each of the patients A to C, or a list of images, vital sign data, estimated results of the condition, etc. for one patient.

[0024] FIG. 3 is a diagram showing the flow of processing performed by the information processing device 3.

[0025] As indicated by arrow A1 in Fig. 3, the information processing device 3 extracts facial feature values ​​and breathing patterns from the facial image. For example, a numerical value indicating a state of distress based on the patient's facial expression is extracted from the facial image as the facial feature value. Also, for example, a numerical value indicating a breathing state based on the movement of the patient's mouth, nose, throat, and neck muscles is extracted from the facial image as the breathing pattern.

[0026] Patients in the ICU are often on ventilators, which obscure part of the patient's face, so using general-purpose facial expression detection techniques to extract facial features can result in poor accuracy.

[0027] Therefore, the information processing device 3 performs facial expression recognition that is specialized in extracting the feature amount around the patient's eyes as the facial feature amount.

[0028] FIG. 4 is a diagram showing the flow of a method for extracting feature amounts around the eyes.

[0029] As indicated by an arrow A21 in Fig. 4, the information processing device 3 roughly detects an area showing the upper half of the patient's face from the facial image. In the example of Fig. 4, as indicated by a rectangular frame F1, the area around the eyes, which surrounds the patient from the nose to the forehead, is detected as the area to be used for extracting feature amounts around the eyes.

[0030] The information processing device 3 cuts out the area around the eyes from the face image to generate a partial image. After rotating the partial image around the eyes, the information processing device 3 detects landmarks around the eyes from the image, as indicated by arrow A22. For example, at least one of the positions of the eyelid edge, the center position of the eye (center position of the iris), the eyebrow position, the inner corner of the eye, the outer corner of the eye, and the bridge of the nose is detected as the position of the landmark around the eyes. Gray dots on the partial image around the eyes indicate the positions of the landmarks around the eyes.

[0031] By detecting landmarks only in the area around the eyes, it is possible to detect landmarks with high accuracy without being affected by the ventilator.

[0032] As indicated by arrow A23, the information processing device 3 extracts eye feature quantities, such as the distance between the inner corners of the eyebrows, the degree of eyelid opening, the number of times the eyelids open and close, the amount of drooping of the outer corners of the eyes, and the direction of gaze, based on the positions of the landmarks around the eyes. These eye feature quantities become numerical values ​​that indicate the distress, depression, energy, etc., felt by the patient. Note that information indicating the relative positional relationship of the landmarks around the eyes may also be used as the eye feature quantities.

[0033] In this way, the information processing device 3 can handle the patient's sedation state, facial expression of pain, state of consciousness, sleep state, etc. as numerical values. The information processing device 3 records the feature amounts around the eyes extracted from the facial image. Because the feature amounts around the eyes are recorded rather than the facial image itself, it is possible to realize a patient monitoring system that takes patient privacy into consideration.

[0034] 3, as indicated by arrow A2, the information processing device 3 extracts posture features from the whole-body image. For example, a numerical value indicating an excited state based on the patient's body convulsions or movements is extracted from the whole-body image as the posture features.

[0035] Patients in the ICU are often covered with a futon. Because the futon hides parts of the patient's body, using general-purpose skeletal structure estimation techniques to extract posture features can result in poor accuracy.

[0036] Therefore, the information processing device 3 performs recognition specialized in extracting the feature amounts of the patient's face and shoulders.

[0037] FIG. 5 is a diagram showing the flow of a method for extracting features of the face and shoulders.

[0038] As indicated by an arrow A31 in Fig. 5, the information processing device 3 roughly detects an area showing the patient's upper body from the whole-body image. In the example of Fig. 5, the area surrounded by a rectangular frame F11 is detected as the area used to extract face and shoulder feature amounts.

[0039] The information processing device 3 cuts out the upper body region from the whole-body image to generate a partial image. After generating the upper body partial image, the information processing device 3 detects the face orientation and shoulder position from the upper body partial image, as indicated by arrow A32. The dashed square on the upper body partial image indicates the patient's face orientation. Furthermore, two gray ellipses indicate the shoulder positions.

[0040] By detecting the shoulder position only in the upper body region, it is possible to detect the shoulder position with high accuracy without being affected by the futon.

[0041] As indicated by arrow A33, information processing device 3 extracts posture feature amounts such as shoulder position, distance between shoulders, angle between shoulders, face direction, etc. Specifically, based on the shoulder position and face direction, posture feature amounts are calculated as numerical values ​​such as the angle at which the body rotates to the left relative to a supine position, the angle at which the face is tilted relative to the shoulder, and the angle at which the right shoulder is raised relative to the left shoulder.

[0042] In this way, the information processing device 3 can handle the patient's sedation state, consciousness state, sleep state, etc. as numerical values. The information processing device 3 records posture feature amounts extracted from whole-body images. Because posture feature amounts are recorded instead of whole-body images, it is possible to realize a patient monitoring system that takes patient privacy into consideration.

[0043] As shown by the dashed line in FIG. 3, the facial feature amounts, breathing pattern, and posture feature amounts extracted from the video as described above are used as appearance feature amounts of the patient in subsequent analysis.

[0044] As indicated by arrow A3, the information processing device 3 performs multivariate analysis using time-series data of appearance features obtained from video images over a predetermined period in addition to the medical data and vital sign data.

[0045] 6 is a diagram showing an example of time-series data used in multivariate analysis, in which the horizontal axis represents time and the vertical axis represents vital sign values ​​or appearance feature amounts.

[0046] As shown in Figure 6, the time-series data of blood test results taken over a specified period, the time-series data of vital signs detected over a specified period, and the time-series data of appearance features extracted from frame images constituting video of a specified period are used in the multivariate analysis.

[0047] If the sampling rates of the vital signs, frame images, blood test results, etc. are different, the information processing device 3 performs interpolation processing to generate vital signs, appearance features (frame images), blood test results, etc. at the same time. The time-series data after the interpolation processing is used for multivariate analysis.

[0048] Multivariate analysis is performed using techniques such as principal component analysis, machine learning, and deep learning. For example, by inputting time-series data of medical records, vital sign data, and appearance features, a predicted value of the vital sign a predetermined time after a reference time is output from the learning model. Examples of the reference time include the time when the frame image is captured, the time when the vital sign is detected, or the time when a blood test is performed. As predicted values ​​of the vital sign, for example, blood pressure, SpO2, heart rate, and respiratory rate after a predetermined time are estimated by multivariate analysis. Furthermore, the probability of a decrease in blood pressure, probability of SpO2, probability of an increase in heart rate, and probability of an increase in respiratory rate may also be estimated by multivariate analysis. In this way, the patient's future condition is estimated by multivariate analysis.

[0049] After performing the multivariate analysis, the information processing device 3 performs an analysis using the attribute data and the results of the multivariate analysis, as indicated by the arrows A4 and A5 in Fig. 3. This analysis is also performed using techniques such as principal component analysis, machine learning, and deep learning.

[0050] Specifically, two patterns of processing can be considered for analysis using attribute data and the results of multivariate analysis.

[0051] In the first pattern of processing, the predicted value of the vital sign after a predetermined time as a result of the multivariate analysis is corrected based on the attribute data, and it is determined whether the corrected predicted value exceeds a threshold value.

[0052] In the second pattern of processing, the threshold is adjusted based on the attribute data, and then a determination is made as to whether the predicted value of the vital sign after a predetermined time as a result of the multivariate analysis exceeds the adjusted threshold.

[0053] After performing an analysis using the attribute data and the results of the multivariate analysis, the information processing device 3 displays abnormality values, predicted abnormality values, physical activity values, etc. on the monitor 4, as indicated by arrow A6. The abnormality values ​​are, for example, values ​​indicating the degree of danger when vital signs suddenly change or the probability of a sudden change in vital signs. The predicted abnormality values ​​are predicted values ​​that exceed a threshold. The physical activity values ​​are values ​​indicating the degree of movement of the patient at the time the frame image is captured. The physical activity values ​​indicate whether the patient is moving, lethargic, etc. Information such as abnormality values, predicted abnormality values, and physical activity values ​​is obtained by analysis using the attribute data and the results of the multivariate analysis.

[0054] As indicated by arrow A7, the information processing device 3 controls the monitor 4 to issue an abnormality alert based on the abnormal numerical value and the predicted abnormal value. For example, if it is determined that the abnormal numerical value or the predicted abnormal value exceeds a threshold, the predicted time of the sudden change in the vital signs and the type of sudden change are displayed on the monitor 4, and an alert is issued to the medical professional warning of the sudden change in the patient's condition. As the type of sudden change, for example, at least one of a drop in blood pressure, a drop in SpO2, an increase in heart rate, and an increase in respiratory rate is displayed on the monitor 4.

[0055] FIG. 7 is a diagram showing an example of a display screen.

[0056] As shown on the left side of FIG. 7, the monitor 4 displays time-series data of vital signs and appearance features, as well as tags T1 to T3 in time series.

[0057] For example, a tag is set for the time when a change in the patient's condition is detected based on the appearance feature, such as when the patient makes a facial expression of distress, opens or closes their eyes, struggles, has a convulsion, etc. A tag is also set for the time when a change in the patient's condition is detected based on vital signs, such as when the patient's blood pressure drops or the respiratory rate increases.

[0058] The video captured around the time when the tag was set is recorded by the information processing device 3. When a tag is selected by a medical professional looking at the display on the monitor 4, the information processing device 3 causes the monitor 4 to display the video of the patient around the time when the tag was set. By selecting the tag, the medical professional can check the video around the time when the patient's condition changed.

[0059] Since video is only recorded when a patient's condition changes, it is possible to reduce the amount of video data storage required.In addition, medical professionals can efficiently check the patient's condition around the time of the change in their condition without having to perform cumbersome operations such as manipulating the video timeline to check the patient's condition.

[0060] It should be noted that tags may be set not only for the time when a change in the patient's condition is detected, but also for future times when a sudden change in vital signs is estimated based on the analysis results by the information processing device 3. The tags set for future times may be displayed on the monitor 4 together with predicted values ​​of the vital signs, for example.

[0061] In this way, the monitor 4 displays information indicating the patient's current condition, such as time-series data on vital signs and appearance features, and information indicating the patient's future condition, such as predicted values ​​of vital signs and alerts.

[0062] As described above, the patient monitoring technology of this invention can quantify the discomfort that medical professionals have empirically judged by observing the patient's appearance based on video footage of the patient, display the quantified appearance features, and predict sudden changes in the patient's condition based on the appearance features.

[0063] Medical professionals can appropriately monitor the state of patients and signs of sudden changes in their condition by looking at the appearance features of each patient displayed on the monitor 4, without having to check on each patient's condition. This reduces the amount of frequent monitoring work required by medical professionals. It also makes it possible to prevent overlooking abnormalities occurring in patients.

[0064] Since the information processing device 3 estimates the patient's condition based on constantly acquired images, vital sign data, and medical data, it becomes possible to constantly monitor the patient's condition, such as 24 hours a day or 365 days a year.

[0065] FIG. 8 is a diagram showing an example of the arrangement of the cameras 1 for each patient.

[0066] As shown in the upper part of Fig. 8, a two-axis (X-axis, Y-axis) rail 11 is fixed to the ceiling near the bed used by the patient, and a camera 1 is provided on the rail 11. The position of the camera 1 can be changed along the rail 11, which serves as a moving mechanism. The rail 11 may also be fixed to the bed used by the patient.

[0067] For example, in an ICU, a patient's posture may change due to changes in the reclining angle of the bed or changes in position. Because the patient's posture changes, it is difficult to always capture an image of the patient's face from the front. Therefore, in the patient monitoring system of this technology, the camera 1 is designed to move to a position where it can capture an image of the patient's face from the front. The position of the camera 1 is controlled by the information processing device 3.

[0068] When the camera 1 captures an image of the patient's face, the camera 1 first captures an image at a low magnification to obtain an image of the patient's entire body from a bird's-eye view. The information processing device 3 detects the position and orientation of the patient's face from the image thus obtained.

[0069] Based on the detection result of the position and orientation of the patient's face, the information processing device 3 moves the camera 1 to a position where it can capture an image of the patient's face from a direction close to the front. Next, the information processing device 3 pans, tilts, and zooms the camera 1 so that the patient's face is captured.

[0070] By performing such control, facial images are acquired. Even if the reclining angle of the bed used by the patient changes, or the patient's posture changes due to positioning, such as lying on their back, facing right, or facing left, camera 1 can be moved to a position where it can capture an image of the patient's face from the front, making it possible to acquire an image that makes it easy to extract facial features.

[0071] The rail 11 on which the camera 1 is mounted can be a rail with one or more axes. The shape of the rail 11 can be straight or curved.

[0072] FIG. 9 is a diagram showing another example of the configuration of a patient monitoring system.

[0073] 9 is configured by connecting a remote monitor 12 to an information processing device 3 in addition to a monitor 4 in the ICU. The remote monitor 12 is connected to the information processing device 3 via, for example, wireless communication.

[0074] The remote monitor 12 is a monitor installed outside the ICU, such as in another hospital. The remote monitor 12 displays information similar to that displayed on the monitor 4 under the control of the information processing device 3. A medical professional in a remote location can give instructions to a medical professional in the ICU while checking the predicted values ​​of the patient's vital signs displayed on the remote monitor 12.

[0075] In this way, information indicating the condition of each of a plurality of patients, estimated taking into account the external appearance of the patient, may be displayed in a list on a monitor provided outside the ICU.

[0076] <2. Configuration of information processing device> 10 is a block diagram showing an example of the functional configuration of the information processing device 3. An example of monitoring the condition of one patient will be described below. In reality, the processing by each component of the information processing device 3 is performed for each of multiple patients.

[0077] As shown in FIG. 10, the information processing device 3 includes an image acquisition unit 21, an appearance feature extraction unit 22, a medical data acquisition unit 23, a vital sign data acquisition unit 24, an attribute data acquisition unit 25, an analysis unit 26, and a display control unit 27.

[0078] Image acquisition unit 21 acquires an image of a patient from camera 1. Furthermore, image acquisition unit 21 controls the position, direction, and angle of view of camera 1 based on the image acquired from camera 1. Frame images constituting the image of the patient are output to appearance feature extraction unit 22.

[0079] The appearance feature extraction unit 22 functions as an analysis unit that analyzes a video of a patient and acquires video analysis information indicating the analysis result. As the video analysis information, for example, appearance feature information is extracted from a frame image supplied from the image acquisition unit 21.

[0080] Specifically, the appearance feature extraction unit 22 detects an area from which to extract appearance features from the frame image. For example, the appearance feature extraction unit 22 detects an area around the patient's eyes or an area of ​​the patient's upper body from the frame image. The appearance feature extraction unit 22 extracts facial features and posture features as appearance features from the detected area.

[0081] The time-series data of the appearance feature amount extracted by the appearance feature amount extracting unit is supplied to the analyzing unit 26.

[0082] The medical data acquisition unit 23 communicates with other systems in the ICU to acquire medical data about the patient. The medical data acquired by the medical data acquisition unit 23 is output to the analysis unit 26.

[0083] The vital sign data acquisition unit 24 acquires the patient's vital sign data from the medical device 2 and outputs it to the analysis unit 26.

[0084] The attribute data acquisition unit 25 communicates with other systems in the ICU, other systems in the hospital, etc. to acquire attribute data about the patient. The attribute data acquired by the attribute data acquisition unit 25 is output to the analysis unit 26.

[0085] The analysis unit 26 performs multivariate analysis using the time-series data of appearance features, the medical data, and the vital sign data. Specifically, the time-series data of appearance features, the medical data, and the vital sign data are input into a learning model, and a predicted value of the vital sign after a predetermined time is output. Note that if the sampling rates of the appearance features, blood test results, and vital signs are different, the analysis unit 26 performs an interpolation process, for example, by adjusting the sampling rate of the information with a lower sampling rate among the appearance features, blood test results, and vital signs to the sampling rate of the information with the highest sampling rate, and then performs multivariate analysis. The analysis unit 26 functions as an interpolation unit that performs an interpolation process for the information with a lower sampling rate.

[0086] The analysis unit 26 performs further analysis using the results of the multivariate analysis and the attribute data. Specifically, the predicted value of the vital sign after a predetermined time and the attribute data are input into the learning model, and a determination result as to whether or not the vital sign will suddenly change is output. Along with this determination result, abnormal numerical values, predicted abnormal values, physical activity values, etc. are also output.

[0087] FIG. 11 is a diagram showing an example of a learning data set of a learning model used in each analysis.

[0088] The learning data set shown in A of FIG. 11 includes time-series data of vital signs, facial features, and posture features as input data, and includes time-series data of vital signs as output data.

[0089] In this way, the learning model used in multivariate analysis using time-series data is generated by machine learning using time-series data of vital signs, facial features, and posture features, which are labeled with time-series data of vital signs indicating the patient's condition, as learning data.

[0090] The learning dataset shown in B of Fig. 11 includes predicted vital sign values ​​and attribute data as input data, and includes values ​​to be corrected for the predicted vital sign values ​​as output data. For example, the difference between the predicted and actual vital sign values ​​is used as the value to be corrected for the predicted value.

[0091] In this way, the learning model used in the analysis using the results of the multivariate analysis and the attribute data is generated by machine learning using the attribute data, in which the differences between the predicted values ​​and the actual measured values ​​of vital signs are labeled, and the predicted values ​​as the results of the multivariate analysis as learning data. The learning model used in the multivariate analysis and the learning model used in the analysis using the results of the multivariate analysis and the attribute data are configured as, for example, LSTM (Long Short-Term Memory).

[0092] 10, analysis unit 26 outputs the results of the multivariate analysis and the results of the analysis using the attribute data to display control unit 27. Analysis unit 26 also functions as an estimation unit that inputs time-series data of appearance features, vital sign data, and the like into a learning model to estimate the condition of a patient.

[0093] Display control unit 27 displays information indicating the patient's condition on monitor 4. For example, the analysis results by analysis unit 26, the time-series data of appearance feature amounts, medical care data, vital sign data, and attribute data are displayed as information indicating the patient's condition on monitor 4. In this case, the same data as the time-series data of appearance feature amounts, medical care data, vital sign data, and attribute data supplied to analysis unit 26 are supplied to display control unit 27.

[0094] Furthermore, the display control unit 27 notifies that a sudden change in the patient's vital signs is predicted by issuing an alert or the like in accordance with the analysis result by the analysis unit 26. The display control unit 27 functions as a monitoring unit that monitors the patient's condition based on the analysis result by the analysis unit 26.

[0095] <3. Operation of the information processing device> The processing of the information processing device 3 will be described with reference to the flowchart of FIG.

[0096] In step S1, the attribute data acquisition unit 25 acquires attribute data about a patient from other systems in the ICU or other systems in the hospital.

[0097] In step S2, the medical data acquisition unit 23 acquires medical data from other systems in the ICU.

[0098] In step S3, the vital sign data acquiring unit 24 acquires time-series data of the vital signs detected by the medical device 2 as vital sign data.

[0099] In step S4, the image acquisition unit 21 controls the position, direction, and angle of view of the camera 1.

[0100] In step S5, the image acquisition unit 21 acquires, from the camera 1, frame images that constitute an image showing the patient.

[0101] In step S6, the appearance feature amount extraction unit 22 detects an area from which an appearance feature amount is to be extracted from the frame image.

[0102] In step S7, the appearance feature amount extraction unit 22 extracts appearance feature amounts from the detected region.

[0103] In step S8, the analysis unit 26 performs multivariate analysis using the time-series data of the appearance feature amounts, the vital sign data, and the medical data.

[0104] In step S9, the analysis unit 26 performs an analysis using the results of the multivariate analysis and the attribute data.

[0105] In step S10, the display control unit 27 causes the monitor 4 to display information indicating the condition of the patient according to the analysis result by the analysis unit .

[0106] After displaying the information indicating the patient's condition on the monitor 4, the process returns to step S3, and the subsequent processes are repeated. When the medical data is updated, for example, when a blood test is performed again, the updated medical data is appropriately acquired by the medical data acquisition unit 23.

[0107] As described above, medical professionals can appropriately monitor the state of patients and signs of a sudden change in their condition by looking at the external features of each patient displayed on monitor 4, without having to check the state of each patient.

[0108] <4. Modifications> The sampling rates of frame images, vital signs, blood test results, etc. may be set according to the severity of the patient, thereby minimizing the overall processing cost of the patient monitoring system.

[0109] The camera 1 may be configured as a night vision camera. Appearance features may be extracted from an image acquired by a depth sensor, an image captured by receiving light in the SWIR (Short Wavelength Infra-Red) wavelength band, or an image captured by a thermal camera.

[0110] Time series data of sensing information of a patient acquired using electromagnetic waves such as millimeter waves may be used for analysis by the information processing device 3. For example, time series data of sensing information acquired using electromagnetic waves and indicating the patient's heart rate and breathing may be used for analysis as vital sign data. Also, time series data of sensing information acquired using electromagnetic waves and indicating the patient's posture may be used for analysis as time series data of appearance features.

[0111] Analysis using the results of multivariate analysis and attribute data can not only determine whether a sudden change in vital signs such as a drop in blood pressure, a drop in SpO2, an increase in heart rate, or an increase in respiratory rate will occur, but can also determine whether an event will occur, such as medical personnel intervening or the patient pressing the nurse call button.

[0112] The learning model used to predict the occurrence of an event is generated by machine learning using a learning dataset that includes attribute data labeled with information indicating the occurrence of the event, such as medical intervention, and predicted values ​​of vital signs as a result of multivariate analysis.

[0113] Multivariate analysis may be performed using multiple learning models that output predicted values ​​of blood pressure, SpO2, heart rate, and respiratory rate after a predetermined time. In this case, a list of predicted values ​​of blood pressure, SpO2, heart rate, and respiratory rate after a predetermined time output from each of the multiple learning models is displayed on monitor 4.

[0114] The learning model used in the multivariate analysis may extract integrated features, which are features obtained by integrating time-series data of appearance features, vital sign data, and medical data. In this case, in the analysis using attribute data, the integrated features and the attribute data are input to the learning model, and a determination result as to whether or not the patient's condition will suddenly change is output.

[0115] An alert may be issued not only when the predicted value of a vital sign (the result of multivariate analysis) exceeds a threshold, but also based on statistics of the predicted value of the vital sign. For example, an alert may be issued that indicates that the predicted value of the vital sign is gradually approaching the threshold, taking into account changes in the predicted value of the vital sign over time.

[0116] About Computers The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0117] FIG. 13 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0118] A CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 are interconnected by a bus 204.

[0119] An input / output interface 205 is also connected to the bus 204. An input unit 206 including a keyboard, a mouse, etc., and an output unit 207 including a display, a speaker, etc. are connected to the input / output interface 205. In addition, a storage unit 208 including a hard disk, a nonvolatile memory, etc., a communication unit 209 including a network interface, etc., and a drive 210 that drives removable media 211 are also connected to the input / output interface 205.

[0120] In the computer configured as above, the CPU 201 loads a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executes the program, thereby performing the above-described series of processes.

[0121] The program executed by the CPU 201 is installed in the storage unit 208 by being recorded on, for example, a removable medium 211 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0122] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0123] ·others In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0124] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0125] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0126] For example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network. The multiple devices can be, for example, IP converters, IP switchers, or servers. For example, each IP converter can extract features from signals output by a connected camera or medical device, and the server can aggregate and analyze the features from each IP converter to estimate the patient's condition.

[0127] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0128] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0129] Configuration combination examples The present technology can also be configured as follows.

[0130] (1) an estimation unit that estimates a condition of the patient by inputting vital information indicating the patient's vital signs and video analysis information obtained by analyzing a video of the patient into a first learning model; a monitoring unit that monitors the state of the patient based on the estimation result by the estimation unit; A patient monitoring system comprising: (2) The vital information is time-series data for a predetermined period of time. The patient monitoring system according to (1) above. (3) The video analysis information is time-series data of features of the patient's appearance extracted from the video for a predetermined period of time. A patient monitoring system according to (1) or (2). (4) The feature amount includes at least one of a facial feature amount, a breathing pattern, and posture information. The patient monitoring system according to (3) above. (5) The first learning model is a machine learning model generated by learning using learning data including the vital sign information labeled with the patient's condition and the video analysis information. A patient monitoring system according to any one of (1) to (4). (6) The first learning model is a machine learning model generated by learning using learning data including the vital sign information labeled with the presence or absence of medical professional intervention and the video analysis information. A patient monitoring system according to any one of (1) to (4). (7) The estimation unit estimates a future state of the patient. A patient monitoring system according to any one of (1) to (6). (8) The facial feature amount is a position of the patient's eye landmark in the image or a numerical value based on the position of the patient's eye landmark. The patient monitoring system according to (4) above. (9) The video analysis information is information indicating the direction of the patient's face and the position of both shoulders, or a numerical value based on the direction of the patient's face and the position of both shoulders. The patient monitoring system according to (4) above. (10) The estimation unit inputs attribute data including at least one of the sex, age, and medical history of the patient and the output result of the first learning model into a second learning model to estimate the condition of the patient. A patient monitoring system according to any one of (1) to (9). (11) The estimation unit inputs the patient's medical data, together with the vital sign information and the video analysis information, into the first learning model to estimate the patient's condition. A patient monitoring system according to any one of (1) to (10) above. (12) The first learning model is LSTM A patient monitoring system according to any one of (1) to (11) above. (13) The monitoring unit notifies the patient of a sudden change in the condition of the patient based on the estimation result by the estimation unit. A patient monitoring system according to any one of (1) to (12) above. (14) The monitoring unit displays information including a probability of a sudden change in the patient's condition. The patient monitoring system according to (13) above. (15) The first learning model outputs at least one of the probability of a decrease in blood pressure, the probability of a decrease in SpO2, the probability of an increase in heart rate, and the probability of an increase in respiratory rate after a predetermined time from the time when the frame image of the video is captured. A patient monitoring system according to any one of (1) to (14) above. (16) the estimation unit inputs the vital sign information and the video analysis information into a plurality of first learning models, each of which outputs different types of information indicating the patient's condition, to estimate the patient's condition; and The monitoring unit displays a list of the different types of information. A patient monitoring system according to any one of (1) to (15) above. (17) The monitoring unit notifies the patient of a sudden change in condition based on statistics of output results of the first learning model. The patient monitoring system according to (13) above. (18) and an interpolation unit that performs an interpolation process on information with a lower sampling rate when the sampling rate of the vital information and the sampling rate of the video analysis information differ. A patient monitoring system according to any one of (1) to (17). (19) The monitoring unit generates a tag based on the condition of the patient, and when a sudden change in the condition of the patient occurs, displays the tag in association with the future time estimated by the estimation unit. A patient monitoring system according to any one of (1) to (18) above. (20) the first learning model outputs an integrated feature that integrates the vital information and the video analysis information; The second learning model receives the integrated feature amount and the attribute data as input and outputs information indicating the condition of the patient. The patient monitoring system according to (10) above. (twenty one) The image capturing device further includes a control unit that controls the position, orientation, and angle of view of a camera that captures the image based on the image. A patient monitoring system according to any one of (1) to (20) above. [Explanation of symbols]

[0131] REFERENCE SIGNS LIST 1 camera, 2 medical device, 3 information processing device, 4 monitor, 11 rail, 12 remote monitor, 21 image acquisition unit, 22 appearance feature extraction unit, 23 medical data acquisition unit, 24 vital sign data acquisition unit, 25 attribute data acquisition unit, 26 analysis unit, 27 display control unit

Claims

1. A feature extraction unit that extracts facial features from an area showing the patient's eyes in a first image of the patient, and extracts posture information from an area showing the patient's upper body in a second image of the patient; an estimation unit that inputs vital information indicating a vital sign of the patient, the facial feature amount, and the posture information into a first learning model to estimate a condition of the patient; a monitoring unit that monitors the state of the patient based on the estimation result by the estimation unit; A patient monitoring system comprising:

2. The vital information is time-series data for a predetermined period of time. The patient monitoring system of claim 1 .

3. The facial feature amount is time-series data extracted from the first video for a predetermined period of time, The posture information is time-series data extracted from the second video for a predetermined period of time.

3. A patient monitoring system according to claim 1 or 2.

4. Further comprising a control unit that controls at least one of the position, imaging direction, and angle of view of a camera that acquires the first image and the second image based on the first image and the second image. A patient monitoring system according to any one of claims 1 to 3.

5. The first learning model is a machine learning model generated by learning using learning data including the vital information labeled with the patient's condition, the facial feature amount, and the posture information. A patient monitoring system according to any one of claims 1 to 4.

6. The first learning model is a machine learning model generated by learning using learning data including the vital information labeled with the presence or absence of medical professional intervention, the facial features, and the posture information. A patient monitoring system according to any one of claims 1 to 4.

7. The estimation unit estimates a future state of the patient. A patient monitoring system according to any one of claims 1 to 6.

8. The facial feature amount is a position of the patient's eye landmark in the first image or a numerical value based on the position of the patient's eye landmark.

4. The patient monitoring system of claim 3.

9. The posture information is information indicating the direction of the patient's face and the position of both shoulders, or a numerical value based on the direction of the patient's face and the position of both shoulders.

4. The patient monitoring system of claim 3.

10. The estimation unit inputs attribute data including at least one of the sex, age, and medical history of the patient and the output result of the first learning model into a second learning model to estimate the condition of the patient. A patient monitoring system according to any one of claims 1 to 9.

11. The estimation unit inputs the medical data of the patient together with the vital information, the facial feature amount, and the posture information into the first learning model to estimate the condition of the patient. A patient monitoring system according to any one of claims 1 to 10.

12. The first learning model is LSTM A patient monitoring system according to any one of claims 1 to 11.

13. The monitoring unit notifies the patient of a sudden change in the condition of the patient based on the estimation result by the estimation unit. A patient monitoring system according to any one of claims 1 to 12.

14. The monitoring unit displays information including a probability of a sudden change in the patient's condition.

14. The patient monitoring system of claim 13.

15. The first learning model outputs at least one of the probability of a decrease in blood pressure, the probability of a decrease in SpO2, the probability of an increase in heart rate, and the probability of an increase in respiratory rate after a predetermined time has elapsed since the frame images of the first video and the frame images of the second video were captured. A patient monitoring system according to any one of claims 1 to 14.

16. the estimation unit inputs the vital information, the facial feature amount, and the posture information into a plurality of first learning models that respectively output different types of information indicating the patient's condition, to estimate the patient's condition; The monitoring unit displays a list of the different types of information.

16. A patient monitoring system according to any preceding claim.

17. The monitoring unit notifies the patient of a sudden change in condition based on statistics of output results of the first learning model.

14. The patient monitoring system of claim 13.

18. and an interpolation unit that performs an interpolation process for information with a lower sampling rate when the sampling rate of the vital information, the sampling rate of the facial feature amount, and the sampling rate of the posture information are different.

18. A patient monitoring system according to any preceding claim.

19. The monitoring unit generates a tag based on the condition of the patient, and when a sudden change in the condition of the patient occurs, displays the tag in association with the future time estimated by the estimation unit.

19. A patient monitoring system according to any preceding claim.

20. the first learning model outputs an integrated feature obtained by integrating the vital information, the facial feature, and the posture information; The second learning model receives the integrated feature amount and the attribute data as input and outputs information indicating the condition of the patient.

11. The patient monitoring system of claim 10.

Images