Computer program, monitoring device, and monitoring method

Abstract

To provide a computer program, monitoring device and monitoring method, which enable estimation of a change in condition of a subject causing an abnormal condition.SOLUTION: A computer program provided herein is configured to acquire sensor data from multiple sensors monitoring the condition of a subject, determine whether the subject is in an abnormal condition or not on the basis of the acquired sensor data, and estimate a condition turning point of the subject leading to an abnormal condition on the basis of sensor data from sensors other than the above sensors, including the sensor data of the sensors used to determine that the subject is in the abnormal condition.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a computer program, a monitoring device, and a monitoring method. [Background technology]

[0002] In recent years, the average life expectancy has increased, leading to an aging society and an increase in the number of nuclear families. In an aging society, various problems such as an increase in the number of people requiring care, the aging of caregivers, and an increase in the care burden on caregivers have become prominent, and it is important to consider how to support the elderly.

[0003] Patent document 1 discloses a care support system that installs multiple sensors within a care facility, receives detection information from the multiple sensors, monitors the condition of the care recipient based on the received detection information, and outputs alert information to the caregiver's mobile device when it is determined that care is needed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-204248 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional systems such as that described in Patent Document 1, a notification for support is sent only when a monitored person, such as a care recipient, is in an abnormal state (abnormal state) that requires care. Depending on the content of the abnormal state, it is important to understand the process leading up to the target person becoming in the abnormal state.

[0006] The present invention has been made in consideration of the above circumstances, and has an object to provide a computer program, a monitoring device, and a monitoring method that are capable of estimating a change in a subject's condition that may cause an abnormal state. [Means for solving the problem]

[0007] The present application includes multiple means for solving the above-mentioned problems. As one example, a computer program causes a computer to execute a process of acquiring sensor data from multiple sensors that monitor the condition of a subject, determining whether or not the subject is in an abnormal state based on the acquired sensor data, and estimating a condition change point of the subject that causes the abnormal state based on sensor data from sensors other than the sensor used to determine that the subject is in an abnormal state. Effect of the Invention

[0008] According to the present invention, it is possible to estimate a change in a subject's condition that causes an abnormal state when the subject falls into the abnormal state. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a monitoring system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of subject data stored in a subject DB. [Diagram 3] FIG. 1 shows a portion of the layout of the care facility in which the subject lives. [Figure 4] FIG. 13 is a diagram illustrating an example of a process of estimating a risk of an abnormal state by a server. [Diagram 5] FIG. 13 is a diagram showing an example of a behavioral state. [Figure 6] FIG. 13 is a diagram illustrating an example of a risk of an abnormal state occurring. [Figure 7] FIG. 13 is a diagram illustrating an example of a process for estimating the risk of an abnormal state using a learning model. [Figure 8] FIG. 1 is a diagram illustrating an example of a subject's behavior. [Figure 9] FIG. 13 is a diagram showing an example of the time progression of a subject's risk of falling into an abnormal state until an incident occurs. [Figure 10] FIG. 13 is a diagram illustrating an example of a rule-based method for estimating a state change point. [Figure 11]FIG. 11 is a diagram illustrating an example of a process for estimating a state change point using a learning model. [Figure 12] 11 is a diagram illustrating an example of a configuration of monitoring history data stored in a monitoring history DB; FIG. [Figure 13] FIG. 13 is a diagram illustrating an example of an incident content estimation process performed by a server. [Figure 14] FIG. 13 is a diagram illustrating an example of the contents of an incident. [Figure 15] FIG. 13 is a diagram illustrating an example of an incident content determination process using a learning model. [Figure 16] FIG. 13 is a diagram illustrating an example of a notification of an incident risk. [Figure 17] FIG. 13 is a diagram showing an example of a notification displayed on the watcher terminal device. [Figure 18] FIG. 13 illustrates an example of a processing procedure for estimating the starting point of an incident occurrence by a server. [Figure 19] FIG. 13 illustrates an example of a processing procedure for predicting an incident occurrence by a server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a monitoring system of this embodiment. The monitoring system includes a server 50 as a monitoring device. The monitoring system may include a plurality of sensors 10, 11, 12, ... 1N, and a relay device 20. A subject DB 61 and a monitoring history DB 62 are connected to the server 50. The server 50 is connected to the relay device 20, the watcher terminal device 30, and the monitoring staff terminal device 40 via a communication network 1. A plurality of sensors 10, 11, 12, ... 1N are connected to the relay device 20. The subject DB 61 and the monitoring history DB 62 may be data servers.

[0011] The multiple sensors 10 to 1N include a mat sensor placed on a bed or the like, a door sensor that detects the opening and closing of a door, a human presence sensor that detects the presence or absence of a person and their movement, a camera that can capture video, and the like. The sensor data detected by the multiple sensors 10 to 1N is transmitted to the relay device 20 together with a sensor ID. The sensors are not limited to the above examples. For example, the sensors also include a biometer and an acceleration sensor that are attached to the body of the person to be monitored. The biometer can measure body temperature, sleep data, activity data (number of steps, walking distance, walking balance, etc.), blood pressure, heart rate, blood oxygen concentration, and the like.

[0012] The relay device 20 includes a memory, a communication module, etc., and temporarily stores sensor data from the multiple sensors 10 to 1N and transmits the stored sensor data to the server 50.

[0013] The watcher terminal device 30 is configured with a smartphone, a tablet terminal, a portable PC, or a desktop PC. Watchers include the subject's family, caregivers who care for the subject, and medical professionals.

[0014] The monitoring staff terminal device 40 is configured with a portable PC, a desktop PC, a tablet terminal, a smartphone, etc. The monitoring staff includes staff who manage and operate the multiple sensors 10 to 1N, the relay device 20, and the server 50.

[0015] The server 50 includes a control unit 51 that controls the entire server 50, a communication unit 52, a memory 53, an interface unit 54, and a storage unit 55. The server 50 may be configured with a computer. The functions of the server 50 may be shared among a plurality of servers.

[0016] The control unit 51 may be configured by incorporating a required number of central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), etc. The control unit 51 may also be configured by combining digital signal processors (DSPs), field-programmable gate arrays (FPGAs), etc.

[0017] The interface unit 54 has an interface function with an external device. The interface unit 54 has a function for accessing the subject DB 61 and the monitoring history DB 62. The interface unit 54 can write data to the subject DB 61 and the monitoring history DB 62, and read data from the subject DB 61 and the monitoring history DB 62.

[0018] The communication unit 52 includes a communication module and has a function of communicating with the relay device 20, the watcher terminal device 30, and the monitoring staff terminal device 40 via the communication network 1. Under the control of the control unit 51, the communication unit 52 acquires (receives) sensor data of the multiple sensors 10 to 1N from the relay device 20.

[0019] The storage unit 55 can be configured with a hard disk or a semiconductor memory, and stores a computer program 56 (program product), learning models 57 and 58, and required information.

[0020] The computer program 56 is a monitoring service application program that runs on the server 50. The computer program 56 may be downloaded from an external device via the communication unit 52 and stored in the storage unit 55. Alternatively, the computer program 56 recorded on a recording medium (e.g., an optically readable disk storage medium such as a CD-ROM) may be read by a recording medium reading unit and stored in the storage unit 55. The computer program 56 may be deployed to be executed on a single computer, or on multiple computers located at one site, or distributed across multiple sites and interconnected by a communication network.

[0021] The memory 53 can be configured with a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. A computer program 56 can be loaded into the memory 53, and the control unit 51 can execute the computer program 56. The control unit 51 can execute processing defined by the computer program 56. The processing by the control unit 51 is also processing by the computer program 56.

[0022] The subject DB 61 stores subject data.

[0023] FIG. 2 is a diagram showing an example of the configuration of subject data stored in the subject DB 61. The subject data is composed of the subject ID, name, date of birth, sex, address (home or family address), care records, medical chart, information on the person watching (telephone number, email address, etc.), etc. The care records include the subject's state of need for support or care, and records of care services received from the past to the present. The medical chart includes the subject's medical examination records, treatment records, and medication records from the past to the present. The watchers include the subject's family members and caregivers. The subject is registered when using the monitoring service of this embodiment.

[0024] The monitoring service of this embodiment assumes that the subject is a care recipient residing in a care facility and the watcher is a caregiver including the subject's family, but is not limited thereto. For example, the subject may be an elderly person living alone and the watcher may be a family member living away from the subject's home.

[0025] FIG. 3 is a diagram showing a part of the layout of a nursing home where a subject lives. FIG. 3 shows a part of the layout related to the subject's range of movement (particularly the range of movement at night). The layout includes the subject's bedroom and toilet, etc., separated by a corridor (passageway). A bed is installed in the bedroom. The bedroom is equipped with a mat sensor placed on the bed, a camera, a bedroom door sensor, etc. A camera is installed in the corridor. The toilet is equipped with a human sensor, a toilet door sensor, etc. These sensors are the sensors 10 to 1N described above.

[0026] Next, a method for estimating the risk that a subject will fall into an abnormal state will be described.

[0027] The control unit 51 has a behavioral state identification processing function for identifying the behavioral state of the subject, and a probability estimation processing function for estimating a risk that the subject will fall into an abnormal state. The risk of falling into an abnormal state is the probability of occurrence of the abnormal state.

[0028] 4 is a diagram showing an example of a process of estimating a risk of an abnormal state by the server 50. The control unit 51 identifies a time-series behavioral state of a subject based on time-series sensor data acquired from a plurality of sensors.

[0029] FIG. 5 is a diagram showing an example of a behavioral state. The behavioral state includes at least one of walking state, body posture, movement time (walking time), time spent in a specific place (such as a toilet), biological data of the subject, sleep data, and activity data. The walking state can be quantified using an evaluation index such as walking style and walking balance. For example, the level of the walking state can be evaluated on a five-level scale from 1 to 5, with a higher value indicating a better state. The levels of body posture, movement time (walking time), time spent in a specific place, biological data of the subject, sleep data, and activity data can also be expressed numerically.

[0030] The behavioral state includes at least one of walking state, body posture, movement time, staying time at a given location, and biometric data of the subject.

[0031] The control unit 51 estimates the risk of the subject becoming in an abnormal state (the probability of occurrence of an abnormal state) based on the identified behavioral state. As shown in FIG. 5, the risk of becoming abnormal can be divided into the risk at a specific location and the risk during movement (walking). The risk at a specific location can be expressed as 0 to 100% for the statistical values ​​(average value, median value, etc.) of the levels of the body posture, the time spent at a specific location, the subject's biological data, sleep data, and activity data among the behavioral states. The risk during movement can be expressed as 0 to 100% for the statistical values ​​(average value, median value, etc.) of the levels of the walking state, body posture, movement time (walking time), the subject's biological data, sleep data, and activity data among the behavioral states.

[0032] Fig. 6 is a diagram showing an example of the risk of an abnormal state occurring. The risks of an abnormal state occurring include risks in the bedroom, risks in the toilet, risks while moving to the bedroom, risks while moving to the toilet, etc. The risks in the bedroom and the toilet are risks in the specific locations mentioned above, and the risks while moving to the bedroom and the toilet are risks while moving mentioned above.

[0033] Next, a method for estimating the risk of an abnormal state using the learning model 57 will be described.

[0034] FIG. 7 is a diagram showing an example of a process of estimating a risk of an abnormal state by the learning model 57. The learning model 57 may use a deep learning network such as a recurrent neural network (RNN), a long short term memory (LSTM) network, a support vector machine (SVM), a random forest, or other such techniques. As shown in FIG. 7, the learning model 57 receives time-series sensor data from a plurality of sensors such as a bedroom mat sensor, a bedroom door sensor, a toilet door sensor, a toilet motion sensor, and a camera. The learning model 57 outputs time-series risk data. The risk data is data classified into risks in the bedroom, risks in the toilet, risks while moving to the bedroom, and risks while moving to the toilet.

[0035] The learning model 57 can be learned (generated) by machine learning using time-series sensor data of many monitored subjects that have been converted into big data. Specifically, training data including sensor data of a plurality of sensors that monitor the states of the multiple monitored subjects when the states of the multiple monitored subjects become abnormal, and the risk of the abnormal state (the probability of occurrence of the abnormal state) is collected. The control unit 51 acquires the training data, and learns (generates) the learning model 57 so as to output the risk of the abnormal state occurring when the sensor data of the multiple sensors is input based on the acquired training data. The risk of the abnormal state occurring may be annotated in association with the sensor data. In addition, the training data may include sensor data of monitored subjects who did not become abnormal. Note that the learning (generation) of the learning model 57 may be performed using a learning server different from the server 50, and the learning model 57 generated by the learning server may be acquired from the learning server.

[0036] FIG. 8 is a diagram showing a schematic example of a subject's behavior. As shown in FIG. 8, first, (1) the subject opens the bedroom door, enters the bedroom, lies down on the bed, and goes to sleep. This behavior can be determined from the sensor data of the bedroom door sensor, the mat sensor, and the camera. Next, (2) the subject gets up from the bed and goes to the bathroom. This behavior can be determined from the sensor data of the bedroom door sensor, the bathroom door sensor, and the camera. Next, (3) the subject relieves himself in the bathroom and leaves the bathroom. This behavior can be determined from the sensor data of the bathroom motion sensor and the bathroom door sensor. Next, (4) the subject moves to the bedroom. This behavior can be determined from the sensor data of the bathroom door sensor, the bedroom door sensor, and the camera. Next, (5) the subject lies down on the bed, and goes to sleep. This behavior can be determined from the sensor data of the mat sensor and the camera. Then, the subject slips (falls) off the bed, and an incident (abnormal state) occurs.

[0037] The control unit 51 acquires sensor data from a plurality of sensors that monitor the state of the subject, and determines whether the subject is in an abnormal state based on the acquired sensor data. When an incident occurs as illustrated in Fig. 8, the control unit 51 estimates a starting point (a point where the state of the subject changes) that is the cause of the incident (abnormal state) based on sensor data from other sensors including the sensor data from the sensor used to determine that the subject is in an abnormal state.

[0038] With the above-described configuration, it is possible to estimate the change in the subject's condition that causes the subject to enter an abnormal state.

[0039] FIG. 9 is a diagram showing an example of the time transition of the risk of an abnormal state until an incident occurs in a subject. The time transition of the risk in FIG. 9 is associated with the behavior exemplified in FIG. 8. As shown in FIG. 9, (1) the risk in the bedroom when lying down in bed and sleeping is set to 0%. Next, (2) the risk while leaving the bedroom and moving to the toilet is set to 20%. This is because, for example, while moving to the toilet, it is determined that the subject is walking with a slight stagger. Next, (3) the risk in the toilet from entering to leaving the toilet is set to 50%. For example, it is determined that the time spent in the toilet (staying time) is longer than usual, and there is a possibility of drowsiness or dozing off. Next, (4) the risk while moving from the toilet to the bedroom is set to 70%. This is because it is determined that the subject is walking with a considerable stagger while moving to the bedroom, and that the time spent returning to the bedroom is longer than usual. Next, (5) the risk in the bedroom is set to the same value as the risk while moving to the bedroom, 70%. This is because there is no particular factor that reduces the risk. After that, the subject slips (falls) and an incident (abnormal state) occurs.

[0040] When the estimated risk (occurrence probability) becomes equal to or greater than the first threshold, the control unit 51 estimates it as a state change point (starting point of the occurrence of the incident) of the subject. In the example of Fig. 9, since the first threshold is set to 10%, the state change point of the subject is the timing of moving from the bedroom to the toilet.

[0041] The control unit 51 can store the sensor data acquired from the estimated state change point (starting point) to the time when it is determined that the state is abnormal (time when the incident occurs) in the storage unit 55 or the monitoring history DB 62. The stored sensor data includes image data of the subject. The control unit 51 can output the stored sensor data as a report. The report can include the date and time of the incident occurrence, the date and time of the starting point, the sensor data from the starting point to the time when the incident occurs, the subject's behavioral state in time series, and risk data in time series.

[0042] With the above-mentioned configuration, the data can be used as training data for caregivers and medical professionals, as information for doctors to use in deciding treatment methods for patients, and as evidence regarding accidents caused by incidents.

[0043] A rule-based method may be used to estimate the state change point (starting point). The rule-based estimation method will be described below.

[0044] Fig. 10 is a diagram showing an example of a method for estimating a state change point using a rule base. A rule base performs judgment according to rules written by humans, and specifically, the control unit 51 performs a process for estimating a state change point according to rules written by humans. The example in Fig. 10 shows an overview of the rule base for convenience, but an actual rule base will have many more branches.

[0045] As shown in FIG. 10, data input to the rule-based algorithm includes time-series sensor data from a bedroom mat sensor, a bedroom door sensor, a toilet door sensor, a toilet motion sensor, a camera, and the like. Based on the input data, it is determined whether the subject's sleep state before getting out of bed has a characteristic. The characteristic state includes, for example, an apnea state, a state in which the number of times of mid-sleep awakening is greater than or equal to a predetermined number, and the like. For example, a camera or a biometric data measuring device may be used to detect the sleep state.

[0046] If the sleep state is characteristic, it is determined whether the subject has gotten out of bed, and if so, whether the subject has entered the toilet. If the subject has entered the toilet, it is determined whether the travel time from the bedroom to the toilet (to entering the toilet) is more than xx hours. xx hours can be set to a time longer than the travel time of a normal subject (without signs of abnormality), and if the travel time is more than xx hours, it is the time that the subject is expected to be walking more slowly than usual, for example, staggering a little.

[0047] If the movement time is more than xx hours, it is determined whether the subject has left the bathroom, and if so, whether they have returned to the bedroom. If the subject has returned to the bedroom, it is determined whether the time from when the subject entered the bathroom to when they returned to the bedroom is more than xx hours. xx hours can be set to a time that is longer than the time that a normal subject (without signs of abnormality) would spend in the bathroom and the time it would take to move from the bathroom to the bedroom, and if the time is more than xx hours, it is the time that the subject is expected to be, for example, drowsy in the bathroom or walking with a significant stagger.

[0048] If the time is more than xx hours, it is determined whether an incident has occurred, and if an incident has occurred, the state change point (starting point of the incident occurrence) is estimated. In this case, the state change point is the timing of moving from the bedroom to the bathroom, which is longer than the usual movement time.

[0049] As described above, the control unit 51 can acquire time-series sensor data from multiple sensors that monitor the subject's condition, and estimate the subject's condition change point using a rule base based on the acquired sensor data.

[0050] Moreover, the state change point (starting point) can be estimated using a learning model. The estimation method using the learning model 58 will be described below.

[0051] FIG. 11 is a diagram showing an example of a state change point estimation process by the learning model 58. The learning model 58 can use a method such as a deep learning network such as a recurrent neural network (RNN) or a long short term memory (LSTM) network. As shown in FIG. 11, time-series sensor data is input to the learning model 58 from multiple sensors such as a bedroom mat sensor, a bedroom door sensor, a toilet door sensor, a toilet motion sensor, and a camera. The learning model 58 outputs the state change point of the subject (the starting point of the incident occurrence).

[0052] The learning model 58 can be learned (generated) by machine learning using time-series sensor data of many monitored subjects that have been converted into big data. Specifically, training data including sensor data of a plurality of sensors that monitor the states of the multiple monitored subjects when the states of the multiple monitored subjects become abnormal, and state change points (starting points of incident occurrence) are collected. The control unit 51 acquires the training data, and learns (generates) the learning model 58 so as to output state change points when sensor data of the multiple sensors is input based on the acquired training data. The state change points may be annotated in association with the sensor data. In addition, the training data may include sensor data of monitored subjects in which no incidents have occurred. Note that the learning (generation) of the learning model 58 may be performed using a learning server different from the server 50, and the learning model 58 generated by the learning server may be acquired from the learning server.

[0053] As described above, the control unit 51 acquires sensor data from multiple sensors that monitor the condition of the subject, and when sensor data is input, the control unit 51 inputs the acquired sensor data to the learning model 58 that outputs a state change point, and can estimate the state change point output by the learning model 58 as the state change point of the subject.

[0054] FIG. 12 is a diagram showing an example of the configuration of monitoring history data stored in the monitoring history DB 62. The monitoring history data is composed of a subject ID, sensor data, time-series data of behavioral states, time-series data of incident occurrence risk, and details of incidents. The sensor data is data detected by a plurality of sensors monitoring each subject. The time-series data of behavioral states is time-series data of walking states, body postures, moving times (walking times), time spent in a given place (e.g., a toilet), biological data of the subject, sleep data, activity data, and the like. The sleep data and activity data include past data prior to the day the incident occurred. The time-series data of incident occurrence risk is data indicating at least the time transition of the incident occurrence risk from the aforementioned starting point to the time of incident occurrence. The details of the incident will be described later.

[0055] Although not shown, according to this embodiment, the following usage is also possible. That is, a camera for monitoring the behavior of a subject is installed at a required location, and the camera is operated at all times to capture videos (images). When it is determined that the condition of the subject has become abnormal (incident has occurred), the video captured during a required time (for example, 1 minute, 2 minutes, 5 minutes, etc.) before and after the incident is cut out (extracted), and the cut out video is stored in the monitoring history DB 62.

[0056] As described above, when the control unit 51 determines that the subject is in an abnormal state, it can extract image data within a required time period before and after the time point at which it is determined that the subject is in an abnormal state from the image data (video) of the subject captured in the sensor data, and store the extracted image data in the monitoring history DB 62. It can be read out and used at any time as objective information or evidence when investigating the cause of an incident.

[0057] Second embodiment In the first embodiment, a configuration for estimating the starting point that causes an incident when the incident occurs is described. In the second embodiment, a prediction of the occurrence of an incident and a prediction of the contents of the incident that will occur are described. The configuration of the monitoring system shown in FIG. 1 is the same in the second embodiment.

[0058] The control unit 51 acquires sensor data from a plurality of sensors that monitor the condition of the subject, identifies the subject's time-series behavioral state based on the acquired sensor data, and estimates the subject's risk of incident occurrence (probability of occurrence of an abnormal state) based on the identified behavioral state (see, for example, FIG. 4). The control unit 51 outputs the estimated risk of incident occurrence. The control unit 51 can output the risk of incident occurrence to the watcher terminal device 30. The output timing can be, for example, the timing when the risk of incident occurrence changes. This allows the watcher to know in advance whether the subject is safe or not, and how close the subject is to an incident occurring, before the incident occurs.

[0059] When the estimated risk of an incident (likelihood of occurrence) becomes equal to or greater than the second threshold, the control unit 51 can notify the user that an incident is predicted to occur. The second threshold may be any value that can notify the user that an incident is predicted to occur with a relatively high probability. This allows the watcher to predict the occurrence of an incident of the subject in advance.

[0060] FIG. 13 is a diagram showing an example of an incident content estimation process by the server 50. The control unit 51 identifies the time-series behavioral state of the subject based on time-series sensor data acquired from multiple sensors. For the behavioral state, see FIG. 5. The control unit 51 has an incident content estimation process function, estimates the content of the incident (abnormal state) of the subject based on the identified behavioral state, and outputs the estimated incident content to the watcher terminal device 30. The output timing can be, for example, the timing when the incident content changes. This allows the watcher to know in advance what kind of incident may occur in the future.

[0061] FIG. 14 is a diagram showing an example of the contents of an incident. Incidents (abnormal states) include, for example, a fall, a fall (slipping down), crouching, stopping walking, apnea, and drowsiness. A fall can be determined by image processing of an image taken by a camera installed in a bedroom or a hallway, or by detecting a sudden change in acceleration based on sensor data of an acceleration sensor worn on the body. A fall is a slip from a bed, and can be determined based on sensor data detected by a mat sensor. Crouching, stopping walking, and drowsiness can also be determined based on whether or not the movement or presence of a subject can be detected using an image taken by a camera, sensor data detected by an acceleration sensor, and multiple door sensors and human sensors. Apnea can be determined based on data measured by a biometer.

[0062] The content of the incident includes at least one of a fall, a drop, crouching, cessation of walking, apnea, and drowsiness.

[0063] FIG. 15 is a diagram showing an example of incident content determination processing by the learning model 57. As with the learning model exemplified in FIG. 7, the learning model 57 may use techniques such as a deep learning network such as a recurrent neural network (RNN), a long short term memory (LSTM) network, a support vector machine (SVM), and a random forest. As shown in FIG. 15, time-series sensor data is input to the learning model 57 from a plurality of sensors such as a bedroom mat sensor, a bedroom door sensor, a toilet door sensor, a toilet motion sensor, and a camera. The learning model 57 outputs the content of the incident. Note that, as shown in FIG. 15, the learning model 57 may output time-series risk data.

[0064] The learning model 57 can be learned (generated) by machine learning using time-series sensor data of many monitored subjects that have been converted into big data. Specifically, sensor data from a plurality of sensors that monitor the states of the multiple monitored subjects when the states of the multiple monitored subjects become abnormal, and training data including the contents of the incident (contents of the abnormal state) are collected. The control unit 51 acquires the training data, and learns (generates) the learning model 57 so as to output the contents of the abnormal state when the sensor data of the multiple sensors are input based on the acquired training data. The contents of the abnormal state may be annotated in association with the sensor data. As described in FIG. 7, the training data may include the risk of an abnormal state (the probability of occurrence of the abnormal state) so that the learning model 57 also outputs the risk of an abnormal state. The learning model 57 may be learned (generated) using a learning server different from the server 50, and the learning model 57 generated by the learning server may be acquired from the learning server.

[0065] Figure 16 is a diagram showing an example of a notification of the risk of an incident. In a series of actions of a subject, a mixture of various types of incident risks occurs, as shown in Figure 14. For convenience, Figure 16 extracts the risk of falling and the risk of tripping (slipping) from the various types of incidents, and shows the transition of the risk of occurrence of each incident.

[0066] Fig. 16A shows the transition of the fall risk over time. As shown in Fig. 16A, if the second threshold for whether or not to notify the fall risk is set to 70%, a notification of the predicted occurrence of an incident (fall) is made when the fall risk reaches 70% or more.

[0067] Fig. 16B shows the transition of the risk of falling over time. As shown in Fig. 16B, if the second threshold for whether or not to notify the risk of falling is set to 50%, a predicted incident (fall) occurrence is notified when the risk of falling reaches 50% or more.

[0068] As described above, the control unit 51 acquires sensor data from a plurality of sensors that monitor the state of the subject, identifies the time-series behavioral state of the subject based on the acquired sensor data, and estimates the occurrence probability of the abnormal state of the subject based on the identified behavioral state. Then, when the estimated occurrence probability is equal to or greater than a second threshold, the control unit 51 notifies that the occurrence of the abnormal state is predicted. The notification destination of the incident occurrence prediction is the watcher terminal device 30 of the watcher.

[0069] Furthermore, the control unit 51 can identify the time-series behavioral state of the subject based on the acquired sensor data, estimate the content of the abnormal state of the subject based on the identified behavioral state, and change the second threshold value according to the content of the estimated abnormal state. In the example of FIG. 16, the second threshold value differs depending on whether the content of the incident is "fall" or "tumble". In this way, by changing the threshold value for notifying the subject of the predicted incident occurrence depending on the level of risk of seriousness or urgency of the content of the incident for the subject, it is possible to notify the subject of the occurrence of the incident at a timing according to the risk of seriousness or urgency of the subject.

[0070] FIG. 17 is a diagram showing an example of a notification screen displayed on the watcher terminal device 30. FIG. 17A shows an example of a notification when an incident occurs as described in FIG. 9. As shown in FIG. 17A, the notification screen displays the target person's name and a message such as "Incident xxxx has occurred." Note that instead of displaying the message, or together with the message, a voice message may be output. By operating the "Confirm" icon, the watcher terminal device 30 transmits information indicating that the notification screen has been confirmed by the watcher to the server 50.

[0071] Fig. 17B shows an example of notifying a predicted occurrence of an incident, as described in Fig. 16. As shown in Fig. 17B, the notification screen displays the target person's name and a message such as "Incident xxxx may occur. The risk of occurrence is □%." Instead of displaying the message, or together with the message, the message may be output as audio. By operating the "Confirm" icon, the watcher terminal device 30 transmits information to the server 50 indicating that the notification screen has been confirmed by the watcher.

[0072] FIG. 18 is a diagram showing an example of a processing procedure for estimating the starting point of an incident occurrence by the server 50. The control unit 51 acquires sensor data from multiple sensors (S11), and determines whether or not an incident has occurred (S12). For details of the incident, see FIG. 15. If an incident has not occurred (NO in S12), the control unit 51 continues the processing from step S11 onwards. If an incident has occurred (YES in S12), the control unit 51 identifies the subject's chronological behavioral state based on the acquired sensor data (S13). For behavioral states, see FIG. 5.

[0073] The control unit 51 estimates the probability of occurrence of the incident (occurrence risk) based on the identified behavioral state (S14). See Figs. 5 to 7 for the risk of incident occurrence. The control unit 51 estimates the starting point of the incident occurrence based on the estimated probability (risk) and a first threshold (S15). See Figs. 8 to 9 for the estimation of the starting point. The control unit 51 stores the sensor data from the starting point to the occurrence of the incident (S16), and ends the process.

[0074] Fig. 19 is a diagram showing an example of a processing procedure for predicting the occurrence of an incident by the server 50. The control unit 51 acquires sensor data from a plurality of sensors (S21), and identifies the time-series behavioral state of the subject based on the acquired sensor data (S22). The control unit 51 estimates the content of the incident based on the identified behavioral state (S23). See Figs. 13 and 15 for estimating the content of the incident. The control unit 51 estimates the probability of occurrence of the incident based on the identified behavioral state (S24).

[0075] The control unit 51 changes the second threshold value according to the contents of the incident (S25), and determines whether the estimated occurrence probability is equal to or greater than the second threshold value (S26). If the occurrence probability is not equal to or greater than the second threshold value (NO in S26), the control unit 51 continues the processing from step 21 onwards, and if the occurrence probability is equal to or greater than the second threshold value (YES in S26), the control unit 51 notifies the watcher terminal device 30 that an incident is predicted to occur (S27), and ends the processing.

[0076] As described above, according to the first embodiment, the starting point that causes the occurrence of an incident can be estimated from the sensor data including the recorded video data. Also, by storing the sensor data from the starting point to the time of the incident, it becomes possible to read and use the sensor data at any time as objective information or evidence when investigating the cause of the incident.

[0077] As described above, according to the second embodiment, unlike the conventional method, it is not necessary to predict the occurrence of an incident by the intuition of a care staff member or the like, but the server 50 can predict the occurrence of an incident. Also, it is possible to sequentially notify the risk of an incident occurring, and to notify the predicted occurrence of an incident at an appropriate timing according to the content of the incident.

[0078] (Supplementary Note 1) The computer program causes a computer to execute processes to acquire sensor data from a plurality of sensors monitoring the condition of a subject, determine whether or not the subject is in an abnormal state based on the acquired sensor data, and estimate a condition change point of the subject that is the cause of the abnormal state based on sensor data from sensors other than the sensor used to determine that the subject is in an abnormal state.

[0079] (Supplementary Note 2) The computer program in Supplementary Note 1 causes a computer to execute a process of storing sensor data acquired during a period from the estimated state change point to a point at which the abnormal state is determined.

[0080] (Supplementary Note 3) In the computer program according to Supplementary Note 1 or 2, the sensor data includes image data of the subject.

[0081] (Appendix 4) The computer program in any one of Appendices 1 to 3 causes a computer to execute processing to identify a time-series behavioral state of the subject based on acquired sensor data, estimate a probability of occurrence of an abnormal state of the subject based on the identified behavioral state, and, when the estimated probability of occurrence is equal to or greater than a first threshold, infer it as a state change point of the subject.

[0082] (Supplementary Note 5) In the computer program according to Supplementary Note 4, the behavioral state includes at least one of a walking state, a body posture, a moving time, a staying time at a predetermined place, and biometric data of the subject.

[0083] (Supplementary Note 6) The computer program in any one of Supplementary Note 1 to Supplementary Note 5 causes a computer to execute a process of estimating a state change point of the subject using a rule base based on acquired sensor data.

[0084] (Appendix 7) The computer program in any one of Appendices 1 to 6 causes a computer to execute a process in which, when sensor data is input, the acquired sensor data is input into a learning model that outputs a state change point, and the state change point output by the learning model is estimated as the state change point of the subject.

[0085] (Appendix 8) The computer program causes a computer to execute a process of acquiring sensor data from a plurality of sensors that monitor the condition of a subject, identifying a time-series behavioral state of the subject based on the acquired sensor data, estimating a probability of occurrence of an abnormal condition of the subject based on the identified behavioral state, and outputting the estimated probability of occurrence.

[0086] (Supplementary Note 9) In Supplementary Note 8, the computer program notifies the user that the abnormal state is predicted to occur when the estimated occurrence probability becomes equal to or greater than a second threshold value.

[0087] (Supplementary Note 10) The computer program in Supplementary Note 8 or Supplementary Note 9 causes a computer to execute a process of identifying a time-series behavioral state of the subject based on the acquired sensor data, inferring the content of the abnormal state of the subject based on the identified behavioral state, and outputting the content of the inferred abnormal state.

[0088] (Supplementary Note 11) The computer program in Supplementary Note 9 causes a computer to execute a process of identifying a time-series behavioral state of the subject based on the acquired sensor data, estimating the content of the abnormal state of the subject based on the identified behavioral state, and changing the second threshold value according to the content of the estimated abnormal state.

[0089] (Supplementary Note 12) In the computer program according to any one of Supplementary Note 8 to Supplementary Note 11, the behavioral state includes at least one of a walking state, a body posture, a moving time, a staying time at a predetermined location, and biometric data of the subject.

[0090] (Supplementary Note 13) In the computer program according to any one of Supplementary Note 8 to Supplementary Note 12, the content of the abnormal condition includes at least one of a fall, a drop, crouching, cessation of walking, apnea, and drowsiness.

[0091] (Appendix 14) A computer program, in any one of Appendices 1 to 13, causes a computer to execute a process of acquiring sensor data from a plurality of sensors monitoring the status of each of a plurality of monitored subjects when the status of the monitored subjects becomes abnormal, and training data including an occurrence probability of the abnormal state, inputting the acquired sensor data into a learning model that outputs an occurrence probability of the abnormal state when sensor data from a plurality of sensors is input based on the acquired training data, and outputting an occurrence probability of the abnormal state of the subjects.

[0092] (Supplementary Note 15) The computer program in any one of Supplementary Note 1 to Supplementary Note 5 causes a computer to execute a process in which, when it is determined that the subject is in an abnormal state, image data within a required time before and after the time when it is determined that the subject is in an abnormal state is extracted from image data of the subject among the sensor data, and the extracted image data is stored.

[0093] (Addendum 16) The monitoring device includes a control unit, which acquires sensor data from a plurality of sensors monitoring the condition of the subject, determines whether the subject is in an abnormal state based on the acquired sensor data, and estimates a condition change point of the subject that causes the abnormal state based on sensor data from sensors other than the sensor used to determine that the subject is in an abnormal state.

[0094] (Addendum 17) The monitoring device includes a control unit that acquires sensor data from a plurality of sensors that monitor the condition of a subject, identifies a time-series behavioral state of the subject based on the acquired sensor data, estimates a probability of occurrence of an abnormal condition of the subject based on the identified behavioral state, and outputs the estimated probability of occurrence.

[0095] (Appendix 18) The monitoring method acquires sensor data from a plurality of sensors monitoring a condition of a subject, determines whether or not the subject is in an abnormal state based on the acquired sensor data, and estimates a condition change point of the subject that causes the abnormal state based on sensor data from sensors other than the sensor used to determine that the subject is in an abnormal state.

[0096] (Appendix 19) The monitoring method acquires sensor data from a plurality of sensors that monitor the condition of a subject, identifies a time-series behavioral state of the subject based on the acquired sensor data, estimates a probability of occurrence of an abnormal condition of the subject based on the identified behavioral state, and outputs the estimated probability of occurrence.

[0097] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that cites at least one multiple claim may also be used. [Explanation of symbols]

[0098] 1. Communication Network 10, 11, 12, …, 1N sensors 20 Relay Device 30. Watcher terminal device 40 Monitoring personnel terminal device 50 Servers 51 Control section 52 Communications Department 53 Memory 54 Interface section 55 Storage section 56 Computer Programs 57, 58 Learning Model 61 Target DB 62 Monitoring History DB

Claims

1. Acquire sensor data from a plurality of sensors that monitor the condition of the subject; determining whether the subject is in an abnormal state based on the acquired sensor data; estimating a state change point of the subject that is the cause of the abnormal state based on sensor data of sensors other than the sensor used to determine that the subject is in an abnormal state; A computer program that causes a computer to carry out processing.

2. storing sensor data acquired from the estimated state change point to the time when it is determined that the abnormal state exists; 2. A computer program product according to claim 1, which causes a computer to carry out a process.

3. The sensor data includes image data of the subject.

3. A computer program according to claim 2.

4. Identifying a time-series behavioral state of the subject based on the acquired sensor data; Estimating the likelihood of an abnormal state occurring in the subject based on the identified behavioral state; When the estimated occurrence probability becomes equal to or greater than a first threshold, the occurrence probability is estimated as a state change point of the subject. A computer program product according to any one of claims 1 to 3, which causes a computer to execute a process.

5. The behavioral state includes at least one of a walking state, a body posture, a moving time, a staying time at a predetermined place, and a biological data of the subject.

5. A computer program according to claim 4.

6. estimating a state change point of the subject using a rule base based on the acquired sensor data; A computer program product according to any one of claims 1 to 3, which causes a computer to execute a process.

7. When sensor data is input, the acquired sensor data is input to a learning model that outputs a state change point, and the state change point output by the learning model is estimated as the state change point of the subject. A computer program product according to any one of claims 1 to 3, which causes a computer to execute a process.

8. Acquire sensor data from a plurality of sensors that monitor the condition of the subject; Identifying a time-series behavioral state of the subject based on the acquired sensor data; Estimating the likelihood of an abnormal state occurring in the subject based on the identified behavioral state; Output the estimated occurrence probability. A computer program that causes a computer to carry out processing.

9. When the estimated occurrence probability becomes equal to or greater than a second threshold value, a notification is given that the occurrence of the abnormal state is predicted.

9. A computer program product according to claim 8, which causes a computer to carry out a process.

10. Identifying a time-series behavioral state of the subject based on the acquired sensor data; Inferring the content of the abnormal state of the subject based on the identified behavioral state; Output the details of the estimated abnormal state.

9. A computer program product according to claim 8, which causes a computer to carry out a process.

11. Identifying a time-series behavioral state of the subject based on the acquired sensor data; Inferring the content of the abnormal state of the subject based on the identified behavioral state; changing the second threshold value according to the content of the estimated abnormal state; 10. A computer program product according to claim 9, which causes a computer to carry out a process.

12. The behavioral state includes at least one of a walking state, a body posture, a moving time, a staying time at a predetermined place, and a biological data of the subject. A computer program according to any one of claims 8 to 11.

13. The abnormal state includes at least one of a fall, a drop, a crouching state, a gait stop, an apnea, and a drowsiness. A computer program according to any one of claims 8 to 11.

14. Acquire training data including sensor data from a plurality of sensors monitoring the states of the plurality of monitored persons when the states of the plurality of monitored persons become abnormal, and an occurrence probability of the abnormal state; inputting the acquired sensor data into a learning model that outputs an occurrence probability of an abnormal state when sensor data from a plurality of sensors is input based on the acquired training data, and outputting an occurrence probability of an abnormal state of the subject; A computer program product according to any one of claims 1 to 3 or claims 8 to 11, which causes a computer to carry out a process.

15. When it is determined that the subject is in an abnormal state, image data within a required time before and after the time point at which it is determined that the subject is in the abnormal state is extracted from image data of the subject among the sensor data; storing the extracted image data; A computer program product according to any one of claims 1 to 5, which causes a computer to carry out a process.

16. A control unit is provided, The control unit is Acquire sensor data from a plurality of sensors that monitor the condition of the subject; determining whether the subject is in an abnormal state based on the acquired sensor data; estimating a state change point of the subject that is the cause of the abnormal state based on sensor data of sensors other than the sensor used to determine that the subject is in an abnormal state; Monitoring equipment.

17. A control unit is provided, The control unit is Acquire sensor data from a plurality of sensors that monitor the condition of the subject; Identifying a time-series behavioral state of the subject based on the acquired sensor data; Estimating the likelihood of an abnormal state occurring in the subject based on the identified behavioral state; Output the estimated occurrence probability. Monitoring equipment.

18. Acquire sensor data from a plurality of sensors that monitor the condition of the subject; determining whether the subject is in an abnormal state based on the acquired sensor data; estimating a state change point of the subject that is the cause of the abnormal state based on sensor data of sensors other than the sensor used to determine that the subject is in an abnormal state; Monitoring method.

19. Acquire sensor data from a plurality of sensors that monitor the condition of the subject; Identifying a time-series behavioral state of the subject based on the acquired sensor data; Estimating the likelihood of an abnormal state occurring in the subject based on the identified behavioral state; Output the estimated occurrence probability. Monitoring method.

Images