Watching system, watching method, and watching program

Abstract

To provide a watching system, a watching method, and a watching program that can estimate the purpose of an action taken by a care object person.SOLUTION: A watching system 10 has history acquisition parts 451, 452, and 453 and a purpose estimation part 454. The history acquisition parts 451, 452, and 453 acquire information about the history of actions taken by a care object person 500. The purpose estimation part 454 estimates the purpose of the actions taken by the care object person 500 based on the information about the history of the actions.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a monitoring system, a monitoring method, and a monitoring program. [Background technology]

[0002] In Japan, the standard of living has improved, the sanitary environment has improved, and the level of medical care has improved, all of which have come with the rapid economic growth since the end of the war, resulting in a remarkable increase in the life expectancy of people. This, combined with the declining birth rate, has resulted in an aging society with a high aging rate. In such an aging society, it is expected that the number of people who require nursing care or nursing care due to illness, injury, aging, etc. (hereinafter referred to as "care recipients") will increase.

[0003] Care recipients are admitted to facilities such as hospitals and elderly welfare facilities, where they receive nursing care and care. In such facilities, incidents may occur, such as when a care recipient falls out of bed or trips while walking, injuring themselves, or getting out of bed and wandering around. Such incidents must be dealt with as quickly as possible. Furthermore, if such an incident is left unattended, it could develop into something even more serious. For this reason, nurses, caregivers, etc. (hereinafter referred to as "care staff") regularly patrol the facilities to check on the safety and condition of the care recipients.

[0004] However, the increase in the number of care staff has not kept up with the increase in the number of care recipients, and the nursing and care industries are chronically short of manpower. Furthermore, since the number of care staff is reduced during night and semi-night shifts compared to daytime shifts, the workload per person increases, and there is a demand to reduce the workload. In order to reduce this labor shortage and the burden on care staff, there is a demand for technology that complements nursing and care work. In recent years, technology to monitor care recipients has been researched and developed as a technology to complement nursing and care work.

[0005] For example, not only can a camera installed in a room at a facility capture images of the care recipient so that the care staff can check on the care recipient's condition, but technology has also been developed that can detect the behavior of the care recipient based on the captured images (videos).The care staff then provide care to the care recipient according to the detected behavior.

[0006] In relation to this, the following Patent Document 1 describes a method for predicting a user's intended behavior by predicting a transition from a user's current location to a staying area based on map information including multiple areas and behavioral history information, and outputting information based on the transition prediction. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-43867 Summary of the Invention [Problem to be solved by the invention]

[0008] However, even if it is possible to predict the transition of a care recipient from their current location to their area of ​​residence, it is difficult to provide appropriate care to the care recipient if the purpose of their actions is not understood.

[0009] The present invention has been made in view of the above-mentioned problems. Therefore, an object of the present invention is to provide a monitoring system, a monitoring method, and a monitoring program that are capable of estimating the purpose of an action taken by a care recipient. [Means for solving the problem]

[0010] The above-mentioned object of the present invention is achieved by the following means.

[0011] (1) A monitoring system having a history acquisition unit that acquires information regarding a history of behavior performed by a person receiving care, and a purpose estimation unit that estimates a purpose of the behavior performed by the person receiving care based on the information regarding the history of the behavior.

[0012] (2) The monitoring system described in (1) above, wherein the information regarding the history of the behavior includes at least one of a movement trajectory of the person being cared for due to the behavior, a skeletal movement of the person being cared for due to the behavior, and a mutual relationship between the person being cared for and an object due to the behavior.

[0013] (3) The monitoring system described in (1) or (2) above, further comprising a behavior detection unit that detects the behavior of the care recipient based on an image of the care recipient, and a history generation unit that generates information regarding the history of the behavior based on the detection result of the behavior of the care recipient by the behavior detection unit.

[0014] (4) The monitoring system according to (3) above, further comprising an image acquisition unit for acquiring an image of the person being cared for.

[0015] (5) The monitoring system described in (3) above, further comprising a memory unit that stores the behavior of the person being cared for detected by the behavior detection unit and the purpose estimated by the purpose estimation unit in association with each other.

[0016] (6) The monitoring system described in (5) above, further comprising a statistical value calculation unit that calculates statistical values ​​for a predetermined period of time regarding the behavior stored in the memory unit, and a statistical value output unit that outputs the statistical values ​​calculated by the statistical value calculation unit.

[0017] (7) A monitoring system as described in (6) above, wherein the statistical value is the number or frequency of the behavior of the person being cared for.

[0018] (8) The monitoring system described in (6) above, further comprising a notification unit that compares the statistical value calculated by the statistical value calculation unit with a predetermined reference value and notifies when the comparison result satisfies a predetermined condition.

[0019] (9) A monitoring system as described in (1) or (2) above, wherein the purpose estimation unit estimates that the purpose of the behavior of the person being cared for is unknown when the person being cared for repeats the same behavior for a certain period of time or a certain number of times.

[0020] (10) The monitoring system described in (5) above, wherein the purpose estimation unit estimates that the purpose of the behavior performed by the person being cared for is unknown if the behavior is other than the behavior stored in the memory unit.

[0021] (11) The monitoring system described in (1) or (2) above, further comprising a purpose output unit that outputs the purpose estimated by the purpose estimation unit.

[0022] (12) The monitoring system described in (2) above, wherein the information regarding the history of the behavior further includes information regarding the time period of the behavior and / or the area in which the care recipient stayed or passed through.

[0023] (13) A monitoring method comprising the steps of: (a) acquiring information regarding a history of behavior performed by a person being cared for; and (b) estimating a purpose of the behavior performed by the person being cared for based on the information regarding the history of behavior.

[0024] (14) A monitoring program for causing a computer to execute a process including the steps of: (a) acquiring information regarding a history of behavior performed by the person being cared for; and (b) estimating the purpose of the behavior performed by the person being cared for based on the information regarding the history of the behavior. Effect of the Invention

[0025] According to the present invention, the purpose of the behavior of the care recipient can be estimated based on information about the history of the behavior of the care recipient. This allows the care staff to attribute meaning to the behavior of the care recipient. Therefore, the care staff can understand "why the care recipient behaved in a certain way," which can be useful for the care and rehabilitation of the care recipient. [Brief description of the drawings]

[0026] [Figure 1] 1 is a diagram showing a schematic configuration of a monitoring system according to an embodiment. [Diagram 2] 2 is a schematic block diagram illustrating a configuration of the information processing device shown in FIG. 1. [Diagram 3] 3 is a block diagram showing the functions of a control unit shown in FIG. 2. [Figure 4] 2 is an explanatory diagram for explaining an example of an image captured by the imaging device shown in FIG. 1. [Diagram 5] FIG. 2 is an explanatory diagram showing examples of feature points of a person; [Figure 6] 11 is an explanatory diagram showing an example of feature points estimated by a posture estimation unit of the behavior detection unit; FIG. [Figure 7] FIG. 13 is a schematic diagram of a captured image including a resident leaving the bed and walking through the center of the room. [Figure 8] 1 is a schematic diagram of a captured image including a resident standing in front of a door to a room and about to go outside through the door. [Figure 9] 1 is a table illustrating an example of a definition of an action purpose. [Figure 10] 13 is a schematic diagram illustrating an example of a captured image including a resident leaving the bed and walking in the center of the room. FIG. [Figure 11] 11 is a schematic diagram illustrating an example of a captured image including a resident standing in front of a curtain and about to open the curtain. FIG. [Figure 12] 11 is a table illustrating an example of a definition when the purpose of an action is unknown. [Figure 13] FIG. 13 is a diagram illustrating an example of a tabulation result of intended behavior of residents. [Figure 14A] 1 is a graph illustrating the number of nighttime toilet visits in a day and toilet frequency in a month. [Figure 14B] 1 is a graph illustrating the number of nighttime exits in a day and the frequency of exits in a month. [Figure 14C] 1 is a graph illustrating the number of nighttime wanderings in a day and the frequency of wanderings in a month. [Figure 15]1 is a flowchart illustrating a main process of a control method for an information processing device according to an embodiment. [Figure 16] FIG. 13 is a diagram illustrating an example of output of objective information for an action. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted. In addition, the dimensional ratios in the drawings are exaggerated for the convenience of explanation, and may differ from the actual ratios.

[0028] <Monitoring System 10> Fig. 1 is a diagram showing a schematic configuration of a monitoring system 10 according to an embodiment. The monitoring system 10 is a system used for care including nursing, caregiving, and support. The monitoring system 10 includes an image capturing device 100, a communication network 200, a mobile terminal 300, and an information processing device 400. The information processing device 400 is connected to the image capturing device 100 and the mobile terminal 300 via the communication network 200 so as to be able to communicate with each other.

[0029] <Photographing device 100> The image capturing device 100 is, for example, a near-infrared camera, and is installed at a predetermined position to capture an image of a capturing area overlooked from the predetermined position as a viewpoint. Hereinafter, an image captured by the image capturing device 100 is referred to as a captured image (see, for example, captured image 601 in FIG. 4). That is, the image capturing device 100 can capture an image of the capturing area by irradiating near-infrared rays toward the capturing area using an LED (Light Emitting Device) and receiving the near-infrared light reflected by an object in the capturing area using a CMOS (Complememtary Metal Oxide Semiconductor) sensor. The captured image can be a monochrome image in which each pixel has a reflectance of near-infrared rays. The predetermined position is, for example, the ceiling of the room of the resident 500. The capturing area is, for example, a three-dimensional area including the entire floor of the room. The image capturing device 100 captures the capturing area as a moving image with a frame rate of, for example, 15 fps to 30 fps. The captured image includes a moving image and a still image. The photographing device 100 transmits the photographed image to the information processing device 400 or the like.

[0030] The photographing device 100 may be, for example, a near-infrared camera included in a sensor box. The sensor box is a box-shaped device equipped with a near-infrared camera, a body motion sensor, etc. In this case, the sensor box may be configured to have some or all of the functions of the information processing device 400.

[0031] The body movement sensor is a Doppler shift type sensor that transmits and receives microwaves to and from the bed and detects a Doppler shift of the microwaves caused by the body movement (eg, breathing movement) of the occupant 500 .

[0032] <Communication network 200> A network interface based on a wired communication standard such as Ethernet (registered trademark) may be used for the communication network 200. A network interface based on a wireless communication standard such as Bluetooth (registered trademark) or IEEE802.11 may also be used for the communication network 200. An access point 210 is provided in the communication network 200, and connects the mobile terminal 300, the photographing device 100, and the information processing device 400 so that they can communicate with each other via a wireless communication network.

[0033] <Mobile terminal 300> The mobile terminal 300 receives information on the behavior of the resident 500 as a care recipient from the information processing device 400, and notifies the care staff of the behavior of the resident 500 by displaying the information on a screen or sounding an alarm. The mobile terminal 300 may be configured, for example, as a smartphone. The mobile terminal 300 may be configured to receive and display captured images from the imaging device 100.

[0034] <Information processing device 400> The information processing device 400 is a device that performs calculations and control in the monitoring system 10 that monitors the behavior of the resident 500. The information processing device 400 receives images captured by the image capturing device 100 from the image capturing device 100, estimates the purpose of the behavior of the resident 500 based on the captured images, and outputs the estimated purpose as information (purpose information). In addition, the information processing device 400 transmits the captured images received from the image capturing device 100 to the mobile terminal 300.

[0035] Fig. 2 is a schematic block diagram illustrating the configuration of the information processing device 400 shown in Fig. 1. The information processing device 400 has a storage unit 410, a display unit 420, an input unit 430, a communication unit 440, and a control unit 450. These components are connected to each other via a bus 460.

[0036] The storage unit 410 can be composed of a RAM (Random Access Memory), a ROM (Read Only Memory), and an SSD (Solid State Drive). The RAM temporarily stores programs and data as a working area for the control unit 450. The ROM stores various programs and data in advance. The SSD stores various programs including a monitoring program and an operating system (OS), and various data.

[0037] The display unit 420 has a display (e.g., a liquid crystal display) and displays various information. For example, in this embodiment, the display displays information on the purpose of the behavior of the resident 500 and a graph of statistical values ​​calculated regarding the intended behavior of the resident 500. The intended behavior is a behavior that is given meaning by estimating its purpose.

[0038] The input unit 430 is configured with, for example, a touch panel and various keys, and is used for various operations and inputs by a user (administrator or care staff).

[0039] The communication unit 440 is an interface for communicating with external devices. For communication, a network interface based on standards such as Ethernet (registered trademark), SATA, PCI Express, USB, and IEEE1394 may be used. For communication, a wireless communication interface such as Bluetooth (registered trademark), IEEE802.11, and 4G may be used. The communication unit 440 receives captured images from the photographing device 100.

[0040] The control unit 450 is configured with a CPU (Central Processing Unit) and performs control and calculation processing of each unit of the information processing device 400 according to a monitoring program. For example, in this embodiment, the control unit 450 calculates various indices such as the movement trajectory, walking speed, and unsteadiness of the resident 500 from the captured image captured by the imaging device 100. Other functions of the control unit 450 will be described in detail below.

[0041] Fig. 3 is a block diagram showing functions of the control unit 450 shown in Fig. 2. The control unit 450 functions as an image acquisition unit 451, a behavior detection unit 452, a history generation unit 453, a purpose estimation unit 454, a purpose output unit 455, a statistical value calculation unit 456, a statistical value output unit 457, and a notification unit 458. In this embodiment, the image acquisition unit 451, the behavior detection unit 452, and the history generation unit 453 configure a history acquisition unit. That is, the history acquisition unit acquires information related to the history of the behavior of the resident 500. Details of the information related to the history will be described later.

[0042] [Image Acquisition Unit 451] The image acquisition unit 451 acquires a captured image from the image capturing device 100 via the communication unit 440. The acquired captured image is transmitted to the behavior detection unit 452.

[0043] Fig. 4 is an explanatory diagram for explaining an example of an image captured by the image capturing device 100 shown in Fig. 1. A captured image 601 includes a plurality of objects including a resident 500, a bed 700, a door 710 for entering and exiting the room, a door 720 for entering and exiting the toilet, a television 730, and a curtain 740. The door 710 and the door 720 can be opened and closed by, for example, sliding.

[0044] [Behavior Detection Unit 452] The behavior detection unit 452 detects the behavior of the resident 500 based on an image of the resident 500. The behavior detection unit 452 detects a person as an object in the room, and identifies the detected person as the resident 500. The behavior detection unit 452 also estimates feature points of the resident 500, and estimates the posture of the resident 500 based on the estimated feature points. Furthermore, the behavior detection unit 452 estimates the behavior of the resident 500 based on the estimated posture of the resident 500. Specifically, this is as follows.

[0045] (Object Detection) The behavior detection unit 452 functions as an object detection unit, and detects a person area 501 and an object area 701 from the captured image 601. The person area 501 is an area within a rectangle (shown with a thick frame) that includes a resident 500 in the captured image 601, and the object area 701 is an area within a rectangle (shown with a dashed line) that includes a specific object other than a person (for example, a bed 700).

[0046] The object detection unit detects an area where an object exists on the captured image 601 as an object existence area, and calculates a reliability score for each predetermined category of the object included in the detected object existence area. The reliability score is a likelihood related to the target object. The behavior detection unit 452 can calculate the reliability score by a known technique using a DNN (Deep Neural Network) such as Faster R-CNN.

[0047] The predetermined category may be, for example, a person, a bed, and an appliance. The appliance includes, for example, a walker, a wheelchair, and the like. The object detection unit detects an object presence region having the highest reliability score of the person category as a person region 501. That is, the object detection unit functions as a detection unit that detects a resident 500 as an object (moving object). Similarly, the object presence region having the highest reliability score of a predetermined object category is detected as an object region 701 (for example, an object region of a bed 700) of the category having the highest reliability score.

[0048] Alternatively, the method for detecting the person area 501 may be a background difference method that extracts the difference between the captured image 601 of the detection target and a background image of the captured area that is extracted in advance by the image capturing device 100. Alternatively, a frame-to-frame (time) difference method or the like may be used that extracts the difference between the captured image 601 of the detection target and a previously captured image.

[0049] The person area 501 and the object area 701 detected by the object detection unit are stored in the storage unit 410.

[0050] The mobile terminal 300 plays back a video based on the multiple frames of captured images received from the image capturing device 100, and displays a rectangle (frame) indicating the person area 501 received from the information processing device 400 at a position on the display corresponding to the person area 501.

[0051] (Posture estimation of 500 residents) The behavior detection unit 452 functions as a posture estimation unit, and estimates the posture of the resident 500 from the person area 501. More specifically, the posture estimation unit estimates feature points 520 (hereinafter also referred to as "feature points 520") related to the human body of the resident 500 from the person area 501, and estimates the posture of the resident 500 based on the estimation result of the feature points 520.

[0052] FIG. 5 is an explanatory diagram showing an example of a person's feature points. In FIG. 5, the positions of the feature points 520 are indicated by white circles. Note that thick black lines connecting the feature points 520 indicate bones, and the human skeleton is indicated by a plurality of feature points 520. In the example of FIG. 5, the posture of the resident 500 standing with both arms spread can be recognized from the relative positions of the feature points 520. The feature points 520 include, for example, feature points 520 of the head, neck, shoulders, elbows, hands, waist, thighs, knees, and feet. The feature points 520 may include feature points 520 other than those described above, or may not include any of the above. The posture estimation unit may estimate the feature points 520 of the resident 500 from the person area 501 by a DNN reflecting a dictionary for detecting the feature points 520 from the person area 501. For example, the posture estimation unit may estimate the feature points 520 by a known technique using a DNN such as DeepPose. Details of DeepPose are described in a publicly known publication (Alexander Toshev, et al. “DeepPose: Human Pose Estimation via Deep Neural Networks”, in CVPR, 2014).

[0053] Fig. 6 is an explanatory diagram showing an example of feature points 520 estimated by a posture estimation unit of the behavior detection unit 452. In the example of Fig. 6, the feature points 520 are shown superimposed on the person area 501. The posture estimation unit estimates the posture of the resident 500 based on the estimation result of the feature points 520. For example, the posture estimation unit calculates a likely posture of the resident 500 from the estimated feature points 520 using a hidden Markov model.

[0054] Other methods for recognizing a person's posture include, for example, a method of determining the posture that has the smallest distance between a predetermined posture model and each feature point position, and a method of learning the posture as a feature and recognizing the posture using the learning results.

[0055] (Estimated behavior of 500 residents) Furthermore, the behavior detection unit 452 functions as a behavior estimation unit, and estimates the behavior of the resident 500 based on the estimated posture of the resident 500 (the positions of the multiple feature points 520). More specifically, the behavior estimation unit estimates the behavior of the resident 500 based on the positional relationship between the multiple feature points 520, the relationship between the positions of the feature points 520 and the position of the object area 701, changes over time in the positions of the multiple feature points 520 (skeletal movement), and the like.

[0056] The positions of the feature points 520 may be represented by relative positions (position coordinates) from a predetermined reference position. For example, the reference position may be the center position of the room of the resident 500. The behavior estimation unit also determines the area (location) where the resident 500 is located based on area information of the room. The area information is, for example, information on the range of each area (for example, the upper left and lower right coordinates of the area) obtained by dividing the room into a plurality of areas according to areas where equipment, fixtures, etc. are located. Examples of the areas include an area where the bed 700 is located, an area by the window, an area at the entrance, an area for the toilet, etc. The area information is, for example, registered in advance in the storage unit 410.

[0057] The actions of the resident 500 include, for example, getting up, sitting, standing up, getting out of bed, entering bed, walking (moving), leaving, transferring, etc. Getting up is the action of getting up from the bed 700. Sitting is the action of sitting on the bed 700 when the resident 500 is in the bed 700 after getting up. The sitting position can be determined based on the skeletal movement (including stopping) of the resident 500 at the edge of the bed 700. Standing up is the action of standing up from the edge of the bed 700 onto the floor. Standing up can be determined based on, for example, the skeletal movement (including stopping) of the resident 500 at the edge of the bed 700.

[0058] Leaving bed is an action of leaving the bed 700. Entering bed is the opposite of leaving bed, and is the action of entering the bed 700. Walking is an action of walking and moving around the room. Walking can be determined, for example, based on the moving speed and skeletal movement of the resident 500. Leaving bed is an action of leaving the room, i.e., leaving the imaging area. Leaving bed includes, for example, going out and going to the toilet. Transferring bed is, for example, an action of transferring from the bed 700 to a wheelchair or the like. Transferring bed can be determined, for example, based on the rotation angle of the trunk of the resident 500 and the distance from the wheelchair.

[0059] Actions such as getting up, sitting, standing up, getting out of bed, entering bed, walking (moving), exiting, transferring, etc. are actions that the resident 500 performs of his / her own volition. On the other hand, actions of the resident 500 also include actions that the resident 500 does not intend, such as falling off the bed 700 (falling) or collapsing onto the floor or the like (tipping).

[0060] The storage unit 410 stores personal information (resident ID or name) that identifies the resident 500, the time when the behavior was detected, and the content of the detected behavior, in association with each other. Hereinafter, information in which the personal information, time, and content of the behavior are associated with each other will be referred to as "behavior information."

[0061] [History Generation Unit 453] Fig. 7 is a schematic diagram of a photographed image including a resident who has left the bed and is walking through the center of the room, and Fig. 8 is a schematic diagram of a photographed image including a resident who is standing in front of the door of the room and is about to go outside through the door.

[0062] For example, a case will be described in which resident 500 shown in Fig. 4 leaves bed 700 and walks toward door 710. As shown in Fig. 7, resident 500 leaves bed 700 and walks through the center of the room. As shown in Fig. 8, resident 500 then moves to the door of the room, stands in front of door 710, and is about to go outside through door 710. In Fig. 8, gray area D1 indicates the space outside the room photographed through the open door 710.

[0063] The history generating unit 453 generates information on the behavior history of the resident 500 based on the multiple feature points 520 of the resident 500 detected by the behavior detecting unit 452, the behavior information of the resident 500, and the area information. The information on the behavior history may include (1) time period, (2) trajectory information, (3) skeletal movement, and (4) interrelationships with objects.

[0064] Among these, the (1) time period may be a time period such as early morning, morning, afternoon, evening, night, or late night when the resident 500 was active. The time period is determined, for example, based on whether or not the time when the activity was detected (the time when the photographed image was taken) is within a range of pre-set time periods such as early morning, morning, afternoon, evening, night, or late night.

[0065] In addition, (2) the trajectory information may be, for example, a movement trajectory TR1 of the person area 501 (that is, the resident 500). The trajectory information also includes information on the areas (places) where the resident 500 stayed or passed through.

[0066] Additionally, (3) skeletal movement is detected by the movement of multiple feature points 520 of the resident, as described above.

[0067] Furthermore, (4) the interrelationship with an object may be, for example, the interrelationship between the resident 500 and an object such as the bed 700, the television 730, or furniture (not shown) such as a chest of drawers. The position of the object may be the position of a rectangle detected by object detection. Furthermore, if the position of the object is fixed, the position of the object may be stored in advance in the storage unit 410. The interrelationship includes the orientation of the resident 500 with respect to the object, the distance between the resident 500 and the object, contact between the resident 500 and the object, interaction between the resident 500 and the object, and the like.

[0068] The movement trajectory, skeletal movement, and interrelationships with objects can be acquired based on the captured images. The information on the behavior history includes at least one of the movement trajectory, skeletal movement, and interrelationships with objects. The information on the behavior history may also include information on the time period of the behavior and / or the area where the resident 500 stayed or passed through.

[0069] [Purpose estimation unit 454] (When the action has a purpose) Fig. 9 is a table showing an example of a definition of the purpose of the action. Fig. 10 is a schematic diagram showing an example of a captured image including a resident leaving the bed and walking in the center of the room. Fig. 11 is a schematic diagram showing an example of a captured image including a resident standing in front of a curtain and about to open the curtain.

[0070] The purpose estimation unit 454 estimates the purpose of the behavior of the resident 500 based on information related to the behavior history. For example, a case will be described in which the resident 500 is sitting on the bed 700 and watching a television 730 as shown in FIG.

[0071] 9, for example, when the behavior detection unit 452 detects a "sitting position," there are two possibilities for the purpose of the behavior. That is, the purpose of the behavior can be either "1. Watching TV" or "8. Changing clothes."

[0072] The purpose estimation unit 454 acquires information on the behavior history that the resident 500 (3) maintains the same posture for a certain period of time (such as sitting on the edge of a chair) and (4) faces the television. From this, the purpose estimation unit 454 estimates that the purpose of the behavior of the resident 500 is "watching television."

[0073] Also, a case will be described assuming that resident 500 shown in Fig. 4 leaves bed 700 and walks toward curtain 740. As shown in Fig. 10, resident 500 leaves bed 700 and walks toward the center of the room. As shown in Fig. 11, resident 500 then moves to the curtain in the room and stands in front of it, intending to open the curtain.

[0074] 9, for example, when "walking (moving)" is detected by the behavior detection unit 452, there are six possibilities for the purpose of the behavior: That is, the purpose of the behavior may be any one of "3. Get something (far away)", "4. Toilet", "5. Go outside", "7. Tidy up (clothes, etc.)", "8. Change clothes", and "9. Open the curtains".

[0075] The purpose estimation unit 454 acquires information on the behavior history, such as (1) early in the morning, (2) moving toward the window, (3) stretching and shortening the arms in front of the window, and (4) staying in a standing position for a certain period of time. From this, the purpose estimation unit 454 estimates that the purpose of the behavior of the resident 500 is to "open the curtains."

[0076] The storage unit 410 stores the behavior detected by the behavior detection unit 452 and the purpose estimated by the purpose estimation unit 454 in association with each other. Hereinafter, the behavior whose purpose is estimated by the purpose estimation unit 454 is referred to as a purpose behavior. In other words, the purpose behavior stored in the storage unit 410 is a behavior whose meaning is given by the purpose estimation unit 454 being estimated.

[0077] As described above, various indicators such as the resident's movement trajectory, walking speed, and unsteadiness within the facility are calculated and are accumulated daily as data representing the resident's characteristics and condition. However, the calculated indicators and detected behaviors alone may not be enough to grasp the purpose of the resident's behavior. By being presented with the purpose information of the resident's behavior, the care staff can understand "why the resident behaved in a certain way." As a result, the care staff can use the presented purpose information to help with the care and rehabilitation of the resident.

[0078] It should be noted that the table shown in FIG. 9 is an example, and the care staff can add or update the definition of the purpose of the action.

[0079] (When the purpose of the action is unclear) FIG. 12 is a table illustrating an example of a definition when the purpose of an action is unknown.

[0080] Resident 500 does not necessarily behave with a clear purpose. For example, if resident 500 suffers from dementia, the surrounding symptoms (BPSD: Behavioral and Psychological Symptoms of Dementia) Dementia may make the purpose of actions unclear.

[0081] As shown in FIG. 12, the behavior detection unit 452 detects "repeated behavior." The purpose estimation unit 454 acquires information on the behavior history, such as (2) visiting the same area multiple times within a certain period of time, and (3) repeating the same movement multiple times within a certain period of time. As a result, the purpose estimation unit 454 estimates that the purpose of the behavior of the resident 500 is unknown ("Unknown 1"). In addition, "Unknown 2" and "Unknown 3" are also estimated in a similar manner. In addition, the purpose estimation unit 454 may estimate that the purpose of the behavior of the resident 500 is unknown if the behavior is other than the behavior stored in the memory unit 410.

[0082] Note that the table shown in FIG. 12 is an example, and the care staff can add or update the definition when the purpose of the action is unknown.

[0083] [Objective Output 455] The purpose output unit 455 outputs the purpose of the behavior of the resident 500 estimated by the purpose estimation unit 454. The purpose output unit 455 controls the display unit 420 to display the purpose on the display. In addition, the purpose output unit 455 can be configured to display the purpose of the behavior on the display of the mobile terminal 300.

[0084] [Statistics Calculation Section 456] FIG. 13 is a diagram illustrating an example of the tabulation result of the intended behavior of a resident. The statistical value calculation unit 456 calculates statistical values ​​for a predetermined period regarding the intended behavior stored in the storage unit 410. The predetermined period may be, for example, one day, one week, two weeks, one month, etc. The statistical value may be the number of times or frequency of the intended behavior of the resident 500. The statistical value calculation unit 456 can tabulate the frequency or time of the relevant intended behavior in a predetermined period (for example, one day). This allows the care staff to grasp the condition of the resident 500.

[0085] Furthermore, the statistical value calculation unit 456 can also tally up behaviors of the resident 500 whose purpose is unclear. For example, the statistical value calculation unit 456 tally up repeated behaviors (repeated same behaviors) of the resident 500 and calculate the number of repeated behaviors in a predetermined period. This allows the care staff to grasp peripheral symptoms of dementia of the resident 500.

[0086] [Statistics output section 457] Fig. 14A is a graph showing the number of nighttime toilet visits in one day and the frequency of toilet visits in one month. Fig. 14B is a graph showing the number of nighttime exits in one day and the frequency of exits in one month. Fig. 14C is a graph showing the number of nighttime wanderings in one day and the frequency of wandering in one month.

[0087] The statistical value output unit 457 outputs the statistical value for each of the intended behavior or non-intended behavior calculated by the statistical value calculation unit 456 to the display unit 420 or the mobile terminal 300. The statistical value output unit 457 outputs, for example, statistical values ​​such as the number of times the resident 500 went to the toilet, the time spent watching television, and the time spent changing clothes, as numerical values, graphs, tables, and the like. In this way, the statistical value output unit 457 can visualize the details of the daily life pattern of the resident 500. By visualizing the records of the intended behavior accumulated daily, the care staff can grasp the intended behavior that the resident 500 can no longer perform. For example, for each intended behavior, a reference value for a certain period is set, and the number, frequency, time, and the like of the intended behavior of the resident 500 are compared with a predetermined reference value (for example, an average value, an upper limit value, or a lower limit value). The condition of the resident is judged according to whether the result of the comparison exceeds or falls below the reference value. For example, if the time required to get changed (purposeful behavior) exceeds an upper limit for a certain period of time, this suggests that the degree of independence of the resident 500 may be declining. This allows the care staff to notice changes in the condition of the resident 500 at an early stage.

[0088] For example, as shown in FIG. 14A, if the number or frequency of nighttime toilet visits is high, there may be a problem with the sleep or health of the resident 500. By displaying the number of nighttime toilet visits as a line graph on a daily basis, the care staff can easily check the number of nighttime toilet visits and whether there is an increasing / decreasing trend. Also, by displaying the toilet frequency on a monthly basis as a bar graph, the care staff can easily check the times of day when the resident 500 frequently goes to the toilet.

[0089] 14B and 14C, if the number or frequency of leaving or wandering at night is high, this may indicate problems with the sleep or health of the resident 500. By displaying the leaving or wandering in a graph on a daily or monthly basis, the care staff can easily check the number of times that leaving / wandering at night occurs, whether there is an increasing / decreasing trend, and the time period of the night-time leaving / wandering.

[0090] [Notification section 458] The notification unit 458 compares the statistical value calculated by the statistical value calculation unit 456 with a predetermined reference value, and issues a notification when the comparison result satisfies a predetermined condition. For example, if the number of repeated actions in a predetermined period exceeds a certain number, the notification unit 458 transmits the result to the mobile terminal 300 of the care staff member since there is a possibility that the dementia of the resident 500 is progressing. Also, if the number of times the resident goes to the toilet per day is below a lower limit, there is a possibility that there is a problem with the health condition of the resident 500, and the care staff member is notified.

[0091] <Control method of information processing device 400> 15 is a flowchart for explaining the main processing steps of the control method of the information processing device 400 in this embodiment. Each process shown in the figure is realized by the CPU of the control unit 450 executing a monitoring program.

[0092] First, the image acquiring unit 451 acquires an image of the resident 500 (step S101). The image acquiring unit 451 acquires video data captured at the above frame rate by the image capturing device 100. The video data is, for example, data for a total of two minutes, that is, one minute before and after the resident 500 gets out of bed 700.

[0093] Next, the behavior detection unit 452 detects the behavior of the resident 500 (step S102). The behavior detection unit 452 calculates a person reliability score for each object existence region for each captured image (for example, captured images 601 to 603). Then, the behavior detection unit 452 detects the resident 500 by detecting the object existence region with the highest reliability score in the person category as the person region 501.

[0094] Storage unit 410 stores data that specifies person area 501 (for example, if person area 501 is rectangular, the coordinates of its vertices).

[0095] The behavior detection unit 452 detects the behavior of the resident 500 for each of the captured images 601 to 603 acquired by the image acquisition unit 451, for example, and stores the detection result in the storage unit 410 together with the resident ID and the detection time. In the example shown in FIG. 4, the behavior detection unit 452 detects that the resident 500 is sitting (sitting position) from the positional relationship of each of the feature points 520 and the positional relationship between the feature points 520 and the bed 700. In addition, in the example shown in FIG. 7, the behavior detection unit 452 detects that the resident 500 is walking (walking) from the positional relationship of each of the feature points 520 and the movement of the skeleton. Furthermore, in the example shown in FIG. 8, the behavior detection unit 452 detects that the resident 500 is walking (walking) from the positional relationship of each of the feature points 520 and the movement of the skeleton.

[0096] Next, the history generating unit 453 generates information on the behavior history of the resident 500 (step S103). The history generating unit 453 generates information on the behavior history of the resident 500 based on the multiple feature points 520 of the resident 500 detected by the behavior detecting unit 452, the behavior information of the resident 500, and the area information. In the example shown in Fig. 8, the information on the behavior history includes the time period of walking, the movement trajectory of the resident 500 approaching (entering) an entrance / exit, the movement of the skeleton while walking, and the absence of any correlation with objects.

[0097] Next, the purpose estimation unit 454 estimates the purpose of the behavior of the resident 500 (step S104). The purpose estimation unit 454 estimates the purpose of the behavior of the resident 500 based on information on the behavior history in accordance with the definition of the purpose of the behavior in Fig. 9. In the example shown in Fig. 8, the information on the behavior history includes a movement trajectory of the resident 500 approaching (entering) an entrance / exit, so the purpose estimation unit 454 estimates that the purpose of the behavior of the resident 500 is "going outside."

[0098] Next, the purpose output unit 455 outputs the purpose of the action of the resident 500 (step S105). The purpose output unit 455 controls the display unit 420 to display "go outside" as the purpose of the action of the resident 500.

[0099] 15, the control unit 450 acquires an image of the resident 500, detects the behavior of the resident 500, and generates information related to the behavior history. In this way, the control unit 450 acquires the information related to the behavior history. Next, the control unit 450 estimates the purpose of the behavior of the resident 500 based on the information related to the behavior history, and outputs the purpose of the behavior of the resident 500 as purpose information.

[0100] (Modification) 16 is a diagram illustrating an example of output of purpose information by the trained model. In the above example, the purpose estimation unit 454 estimates the purpose of the behavior of the resident 500 based on information on the behavior history in accordance with the definition of the purpose of the behavior in FIG.

[0101] In this modification, instead of the above definition of the purpose of the behavior, a trained model that generates purpose information of the behavior is used to estimate the purpose of the behavior of the resident 500 from information on the behavior history of the resident 500. FIG. 16 illustrates an example of purpose information output by the trained model. The trained model is machine-trained based on training data including information on the behavior history and correct answer data on the purpose information of the behavior, with information on the behavior history as an explanatory variable and the purpose information of the behavior as a target variable, for multiple residents of the facility.

[0102] The monitoring system 10, the monitoring method, and the monitoring program of the present embodiment described above provide the following effects.

[0103] Based on information about the history of the actions taken by the resident 500, the purpose of the actions taken by the resident 500 can be estimated. This allows the care staff to attribute meaning to the actions taken by the resident 500. Therefore, the care staff can understand "why the resident 500 acted in a certain way," which can be useful for the care and rehabilitation of the resident 500.

[0104] The present invention is not limited to the above-described embodiment, but can be modified in various ways within the scope of the claims.

[0105] In the above embodiment, the control unit 450 of the information processing device 400 functions as the image acquisition unit 451, the behavior detection unit 452, the history generation unit 453, the purpose estimation unit 454, the purpose output unit 455, the statistical value calculation unit 456, the statistical value output unit 457, and the notification unit 458. However, the present invention is not limited to such a case. For example, the monitoring system 10 can be configured to further include a server that functions as the image acquisition unit 451, the behavior detection unit 452, and the history generation unit 453, instead of the control unit 450. Therefore, in this case, the control unit 450 functions as the history acquisition unit, the purpose estimation unit 454, the purpose output unit 455, the statistical value calculation unit 456, the statistical value output unit 457, and the notification unit 458.

[0106] Alternatively, the photographing device 100 may be configured to function as an image acquisition unit 451, a behavior detection unit 452, a history generation unit 453, a purpose estimation unit 454, a purpose output unit 455, a statistical value calculation unit 456, a statistical value output unit 457, and a notification unit 458.

[0107] The monitoring program may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a memory, storage, or the like. The image processing program may be provided as a standalone application software, or may be incorporated into the software of each device as a function of the server.

[0108] Furthermore, in the embodiment, a part or all of the processing executed by the image processing program may be replaced with hardware such as a circuit. [Explanation of symbols]

[0109] 10 Monitoring system, 100 imaging device, 200 communication networks, 210 access points, 300 mobile devices, 400 Information processing device, 410 storage section, 420 display section, 430 input unit, 440 Communications Department, 450 control section, 451 Image acquisition unit, 452 behavior detection unit, 453 History Generation Unit, 454 Purpose Estimation Department, 455 target output section, 456 Statistics Calculation Unit 457 Statistics output section 458 Notification Department; 500 occupants, 501,502 people area, 520 feature points, 601~605 Photographed images, 700 beds, 701 object region, 710 Door, 720 doors, 730 TV, 740 Curtains.

Claims

1. a history acquisition unit that acquires information regarding a history of actions taken by a care recipient; A monitoring system having a purpose estimation unit that estimates the purpose of the behavior of the care recipient based on information regarding the behavior history.

2. The monitoring system of claim 1, wherein the information regarding the history of the behavior includes at least one of a movement trajectory of the person being cared for due to the behavior, a skeletal movement of the person being cared for due to the behavior, and a mutual relationship between the person being cared for and an object due to the behavior.

3. A behavior detection unit that detects a behavior of the care recipient based on an image of the care recipient; The monitoring system according to claim 1 , further comprising a history generating unit configured to generate information regarding a history of the behavior of the care recipient based on a detection result of the behavior of the care recipient by the behavior detecting unit.

4. The monitoring system according to claim 3 , further comprising an image acquisition unit that acquires an image of the care recipient.

5. The monitoring system according to claim 3 , further comprising a storage unit that stores the behavior of the care recipient detected by the behavior detection unit and the purpose estimated by the purpose estimation unit in association with each other.

6. a statistical value calculation unit that calculates a statistical value for a predetermined period of time regarding the actions stored in the storage unit; The monitoring system according to claim 5 , further comprising: a statistical value output unit that outputs the statistical value calculated by the statistical value calculation unit.

7. The monitoring system according to claim 6 , wherein the statistical value is the number or frequency of the behavior of the care recipient.

8. The monitoring system according to claim 6 , further comprising a notification unit that compares the statistical value calculated by the statistical value calculation unit with a predetermined reference value and notifies when a comparison result satisfies a predetermined condition.

9. The monitoring system according to claim 1 or 2, wherein the purpose estimation unit estimates that the purpose of the action taken by the care recipient is unclear when the care recipient repeats the same action for a certain period of time or a certain number of times.

10. The monitoring system according to claim 5 , wherein the purpose estimation unit estimates that a purpose of the behavior performed by the care recipient is unknown when the behavior is other than the behavior stored in the memory unit.

11. The monitoring system according to claim 1 , further comprising a purpose output unit that outputs the purpose estimated by the purpose estimation unit.

12. The monitoring system according to claim 2 , wherein the information regarding the history of the behavior further includes information regarding a time period during which the behavior occurred and / or an area in which the care recipient stayed or passed through.

13. (a) obtaining information regarding a history of actions taken by a subject of care; and (b) estimating the purpose of the behavior of the person being cared for based on information regarding the history of the behavior.

14. (a) obtaining information about the history of actions taken by a care subject; A monitoring program for causing a computer to execute a process including a step (b) of estimating the purpose of the behavior performed by the person being cared for based on information regarding the history of the behavior.

Images