Watching system, watching device, watching method, and watching program

Abstract

To provide a watching system capable of suppressing false alarms such as falling over or falling down of a monitored person.SOLUTION: A monitoring system includes: an imaging part configured to captured an imaging region including at least the region around a bed; a sensor configured to detect a load on the bed; a determination part configured to determine a state of a monitored person based on the image and an output from the sensor; and a terminal configured to transmit information regarding the state of the monitored person based on the determination. When it is determined that a predetermined load is detected based on the output from the sensor, the monitoring system performs control not to transmit information on the state of the monitored person even when any of the states of falling over, falling down, and getting out of the bed is detected based on the image.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring device, a monitoring method, and a monitoring program. [Background technology]

[0002] In nursing care facilities such as nursing homes for the elderly and hospitals, residents or hospitalized care recipients (hereinafter referred to as "care recipients") are at risk of falling while walking or falling out of bed and getting injured. Therefore, to enable nurses or caregivers (hereinafter referred to as "caregivers") to rush to the scene when a resident falls into such a state, development is underway to develop a system that can constantly monitor the resident's condition.

[0003] The following prior art is disclosed in Patent Document 1: The area around a bed is photographed from above the floor and at a height higher than the height of the person being monitored to obtain image data, skeletal information is extracted from the image data, and the posture of the person being monitored is detected based on the skeletal information and pre-stored skeletal information samples. Then, based on the detected posture, it is determined whether the person being monitored has tipped over or fallen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-34960 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the prior art disclosed in Patent Document 1, based on the knowledge that falls and trips occur on the floor other than the bed, determines whether or not the monitored person has fallen and tripped based on image data captured in the area surrounding the bed. Therefore, if a caregiver working beside the bed to care for a monitored person who is sleeping on the bed makes a movement such as crouching down, it may be determined that the monitored person has fallen and tripped.

[0006] Furthermore, even if the shooting area is set to an area including the bed, it is difficult to recognize the monitored person as a person from the image because the person is covered with a blanket, and recognition becomes even more difficult if the monitored person is not moving. For this reason, even if the area is set to include the bed, it is not possible to prevent false alarms such as falls and trips due to the posture of a caregiver working near the bed being mistaken for the posture of the monitored person.

[0007] The present invention has been made to solve such problems, and aims to provide a monitoring system, a monitoring device, a monitoring method, and a monitoring program that can suppress false alarms about a monitored person's falling or tripping. [Means for solving the problem]

[0008] The above-mentioned problems of the present invention are solved by the following means.

[0009] (1) A device that includes an imaging unit that captures an image of a photographing area including at least the area surrounding the bed, a sensor that detects a load on the bed, a determination unit that determines the state of the monitored person based on the captured image and the output of the sensor, and a terminal that notifies information about the state of the monitored person based on the determination by the determination unit, and when it determines that a predetermined load has been detected based on the output of the sensor, the device determines based on the image that the monitored person has fallen, fallen down, or gotten out of bed. A monitoring system that performs control so as not to report the information even if the state is detected.

[0010] (2) The monitoring system described in (1) above, wherein the control not to notify is that the judgment unit does not output the information.

[0011] (3) The monitoring system according to (1) above, wherein the control not to report is such that the terminal does not report even if it receives the information.

[0012] (4) The monitoring system described in (1) above, wherein the control of not reporting is such that the judgment unit does not judge the state detected based on the image as the state of the monitored person.

[0013] (5) A monitoring system as described in (2) or (3) above, which has an image memory unit that stores images captured by the imaging unit, and when any of the following conditions is detected based on the images: falling, tripping, or getting out of bed, an image relating to the condition is stored in the image memory unit.

[0014] (6) A monitoring system described in any of (2), (3), and (5) above, which has a judgment result memory unit that stores the judgment result of the judgment unit, and when any of the states of falling, dropping, or getting out of bed is detected based on the image, the result is stored in the judgment result memory unit.

[0015] (7) The monitoring system described in (4) above, wherein the state further includes leaving the room.

[0016] (8) The monitoring system described in (3) above, wherein the judgment unit includes information prohibiting notification in the information and transmits it to the terminal.

[0017] (9) A monitoring system described in any of (1) to (8) above, wherein the judgment unit calculates the difference for each pixel between adjacent frames of the image and judges the state of the monitored person based on a moving silhouette detected by binarizing the difference.

[0018] (10) A monitoring system described in any of (1) to (8) above, wherein the judgment unit estimates joint points from the image using a trained model based on machine learning, and judges the state of the monitored person based on the estimated joint points.

[0019] (11) The monitoring system described in (9) citing (7) above, wherein when the judgment unit determines that the person has left the room, the judgment unit adjusts the threshold value for the binarization to reduce the detection sensitivity of the moving silhouette.

[0020] (12) The monitoring system according to any one of (1) to (11) above, wherein the photographing area is an area including the area on the bed and the area around the bed.

[0021] (13) A monitoring device having an acquisition unit that acquires an image of a shooting area that includes at least the area surrounding the bed and the output of a sensor that detects the load on the bed, a judgment unit that judges the condition of the monitored person based on the image and the output of the sensor, and a notification unit that notifies information regarding the condition of the monitored person based on the judgment by the judgment unit, and when it judges that a specified load has been detected based on the output of the sensor, it performs control not to notify the information even if it detects any of the conditions of falling, dropping, or getting out of bed based on the image.

[0022] (14) The monitoring device according to (13) above, wherein the control not to notify is that the determination unit does not output the information.

[0023] (15) The monitoring device described in (13) above, wherein the control of not reporting is such that the judgment unit does not judge the state detected based on the image as the state of the monitored person.

[0024] (16) A monitoring device as described in (14) above, which has an image memory unit that stores captured images, and when any of the following conditions is detected based on the images: falling, tripping, or getting out of bed, an image relating to the condition is stored in the image memory unit.

[0025] (17) A monitoring device as described in (14) or (16) above, which has a judgment result memory unit that stores the judgment result of the judgment unit, and when any of the states of falling, dropping, or getting out of bed is detected based on the image, the judgment result memory unit stores the result.

[0026] (18) The monitoring device according to (15) above, wherein the state further includes leaving the room.

[0027] (19) The control not to notify is a monitoring device described in (13) above, in which the judgment unit includes information prohibiting notification in the information and transmits information regarding the status of the monitored person based on the judgment by the judgment unit to a terminal that will notify.

[0028] (20) A monitoring method comprising the steps of: (a) acquiring an image of a photographed area including at least the area surrounding the bed and the output of a sensor that detects a load on the bed; (b) determining the condition of the monitored person based on the image and the output of the sensor; (c) notifying information regarding the condition of the monitored person based on the determination in step (b); and (d) performing control in step (c) not to notify the information when it is determined based on the output of the sensor that a predetermined load has been detected, even if any of the states of falling, dropping, or getting out of bed is detected based on the image.

[0029] (21) The monitoring method of (20) above, wherein the control not to notify is to not output the information in step (b).

[0030] (22) The monitoring method of (20) above, wherein the control not to notify is to not judge the state detected based on the image in step (b) as the state of the monitored person.

[0031] (23) The monitoring method described in (21) above, comprising a step (e) of storing an image relating to a state of falling, dropping, or getting out of bed when the state of falling is detected based on the image.

[0032] (24) A monitoring method as described in (21) or (23) above, comprising a step (f) of storing the judgment result in step (b) when any of the states of falling, dropping, or getting out of bed is detected based on the image.

[0033] (25) The monitoring method described in (22) above, wherein the state further includes leaving the room.

[0034] (26) The monitoring method described in (20) above, wherein the control not to notify is, in step (b), including information prohibiting notification in the information, and transmitting information regarding the monitored person's status based on the judgment to a terminal that will notify.

[0035] (27) A monitoring program for causing a computer to execute the monitoring method described in any one of (20) to (26) above. [Effects of the Invention]

[0036] The system determines the condition of the person being monitored based on an image of an area including at least the area around the bed and the output of a sensor that detects the load on the bed, and reports information about the monitored person's condition based on the determination. When a predetermined load is detected by the sensor, the system controls not to report information about the monitored person's condition even if it detects a fall, a slip, or getting out of bed based on the image. This makes it possible to prevent false reports of the monitored person falling, etc. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a diagram showing the overall configuration of a monitoring system. [Figure 2] FIG. 10 is a diagram showing the inside of a room of a care recipient in which a detection unit is arranged. [Figure 3] FIG. 2 is a block diagram showing a hardware configuration of a detection unit. [Figure 4] FIG. 10 is a diagram showing an image captured by a camera. [Figure 5]FIG. 10 is an explanatory diagram showing a comparative example in which a false alarm of a fall occurs. [Figure 6] 10A and 10B are explanatory diagrams showing an embodiment in which false alarms of falls are suppressed. [Figure 7] FIG. 10 is an explanatory diagram showing a comparative example in which a false alarm of bed exit occurs. [Figure 8] FIG. 10 is an explanatory diagram showing an embodiment in which false alarms of bed exit are suppressed. [Figure 9] FIG. 10 is an explanatory diagram showing a comparative example in which a false alarm of leaving a room occurs. [Figure 10] FIG. 10 is an explanatory diagram showing an embodiment in which false alarms of leaving a room are suppressed. [Figure 11] FIG. 2 is a block diagram showing the hardware configuration of a server. [Figure 12] FIG. 10 is a diagram illustrating a management table. [Figure 13] FIG. 2 is a block diagram showing the hardware configuration of the mobile terminal. [Figure 14] 10 is a flowchart showing the operation of the monitoring system. DETAILED DESCRIPTION OF THE INVENTION

[0038] Hereinafter, a monitoring device, a monitoring system, a monitoring method, and a monitoring program according to embodiments of the present invention will be described with reference to the drawings. In the drawings, identical elements are designated by the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0039] FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 according to this embodiment. The monitoring system 1, which is used in nursing care facilities and the like, includes a detection unit 10, a server 20, multiple mobile terminals 30, and a wireless AP (access point) 31. The detection unit 10, server 20, mobile terminals 30, and detection unit 10 are interconnected so as to be able to communicate with each other via a LAN (local area network) and the wireless AP 31. The mobile terminal 30 is carried by a caregiver 70 who nurses or cares for a care recipient 80. The detection unit 10 constitutes a monitoring device.

[0040] (Detection unit 10) FIG. 2 is a diagram showing the interior of a room (hereinafter also referred to as a "living room") of a care recipient 80 in which the detection unit 10 is arranged. FIG. 3 is a block diagram showing the hardware configuration of the detection unit 10. As shown in FIG. 3, the detection unit 10 includes a bed sensor 103 and a camera 104. The bed sensor 103 constitutes a sensor. The camera 104 constitutes an imaging unit.

[0041] As shown in Fig. 2, the bed sensors 103 are placed on the beds 90 in the rooms of the monitored care recipients 80. Specifically, the bed sensors 103 are placed, for example, under the mattresses on the beds 90. The cameras 104 can be placed, for example, on the ceiling or upper part of the walls of the rooms, or attached to the beds 90 (see Fig. 6, etc.).

[0042] As shown in FIG. 3, the detection unit 10 includes a control unit 101, a communication unit 102, and a bed sensor 10 The control unit 101 includes a determination unit, an image storage unit, and a determination result storage unit. The control unit 101, together with the communication unit 102, also includes an acquisition unit and a notification unit.

[0043] The control unit 101 includes a CPU, RAM, ROM, etc. The control unit 101 may include an HDD or SSD. The control unit 101 controls each unit of the detection unit 10 and performs calculation processing according to a program. The functions of the control unit 101 will be described in detail later.

[0044] The communication unit 102 is an interface circuit (for example, a LAN card, a wireless communication circuit, etc.) for communicating with, for example, the server 20 via a LAN.

[0045] The bed sensor 103 outputs a signal corresponding to the load on the bed 90. The bed sensor 103 may be configured, for example, by a sheet-like bag filled with air and a piezoelectric sensor that detects fluctuations in the pressure of this air.

[0046] The camera 104 captures an image of a photographing area including at least the peripheral area 91 of the bed 90 (see FIG. 4 ), and outputs the photographed image 1041 (image data). Hereinafter, the image 1041 output by the camera 104 will also be simply referred to as the "image 1041." The photographing area may be an area including the top of the bed 90 (including a part of the top of the bed 90) and the peripheral area 91 of the bed 90. The photographing area may also be an area including only the peripheral area 91 of the bed 90.

[0047] The image 1041 includes a still image and a moving image. The camera 104 is a near-infrared camera, but a visible light camera may be used instead, or a combination of these may be used.

[0048] Fig. 4 is a diagram showing an image 1041. The image 1041 shown in Fig. 4 is an image 1041 captured by a camera 104 attached to a bed 90. Fig. 4 shows the image 1041 captured with an area including the top of the bed 900 and a peripheral area 91 of the bed 90 as the captured area.

[0049] The function of the control unit 101 will now be described.

[0050] The control unit 101 detects the body movements of the care recipient 80 on the bed 90 based on the output of the bed sensor 103. As a result, the control unit 101 can detect the respiratory rate, heart rate, etc. of the care recipient 80 and transmit the detection results to the server 20. The control unit 101 may further detect whether the care recipient 80 is asleep, awake, etc. based on the output of the bed sensor 103 and transmit the detection results to the server 20.

[0051] The control unit 101 determines the state of the care recipient 80 based on the image 1041 and the output of the bed sensor 103, and transmits (outputs) the determined state to the server 20 as information relating to the state of the person being monitored. The state of the care recipient 80 includes specific states such as falling, tripping, getting out of bed, and leaving the room. The state of the care recipient 80 also includes states other than the specific states such as walking and washing the face.

[0052] The control unit 101 can detect the state of the care recipient 80, for example, by one of the following two methods. In the first method, the control unit 101 calculates the difference for each pixel between frames of the image 1041 and detects a moving silhouette by binarizing the difference (time difference method). The control unit 101 then detects the state of the care recipient 80 based on the moving silhouette. Specifically, the control unit 101 can detect a fall, for example, by the change in the center of gravity of the detected moving silhouette from a state of moving in time series to a state of suddenly stopping, and by a change in the aspect ratio of the rectangle corresponding to the moving silhouette. The control unit 101 can detect a fall, for example, by the change in the center of gravity of the detected moving silhouette from a state of moving in time series to a state of suddenly stopping, and by a change in the aspect ratio of the rectangle corresponding to the moving silhouette. A fall can be detected by a sudden change in the silhouette of the moving body from being within the area of ​​bed 90 to being outside the area of ​​bed 90, and by the width of overlap between the boundary of the area of ​​bed 90 and the moving body silhouette. The area of ​​bed 90 in image 1041 is set in advance when detection unit 10 is installed in each room and can be stored as data in the memory of control unit 101. For example, control unit 101 can detect leaving the bed by the moving body silhouette moving from being within the area of ​​bed 90 to being outside the area of ​​bed 90. For example, control unit 101 can detect leaving the room by the moving body silhouette moving to an area near the door and disappearing there. The area near the door in image 1041 can be stored as data in the memory of control unit 101, similar to the area of ​​bed 90.

[0053] As a second method, the control unit 101 may estimate joint points from the image 1041 using a trained model based on machine learning and detect the state of the care recipient 80 based on the estimated joint points. Specifically, the control unit 101 detects an area where an object exists for each frame of the image 1041 and estimates the category of the object included in the detected area, thereby detecting a person area. The area where the object exists may be detected as a candidate rectangle on the image 1041, which is a rectangle that includes the object. The control unit 101 detects the person area by detecting, from the detected candidate rectangles, a candidate rectangle whose object category is estimated to be a person. The person area may be detected using a neural network. Examples of methods for detecting the person area 610 using a neural network include well-known methods such as Faster R-CNN, Fast R-CNN, and R-CNN. The neural network for detecting a person region from the image 1041 is trained in advance to detect (estimate) a person region from the image 1041 using training data for a combination of the image 1041 and a person region set as a correct answer for the image 1041. The control unit 101 detects articulation points from the person region using a known technique using a neural network, such as DeepPose. The articulation points can be detected as coordinates in the image 1041. The neural network for detecting articulation points from the person region is trained in advance to identify (estimate) articulation points from the person region using training data for a combination of the person region and the articulation points set as a correct answer for the person region. The control unit 101 estimates the posture of the care recipient 80 based on the articulation points. The posture can be estimated using a neural network for detecting posture from the articulation points. In this case, the neural network for detecting posture from the articulation points is trained in advance to estimate a posture from the articulation points using training data for a combination of the articulation points and a posture set as a correct answer for the articulation points. The posture may be estimated using a hidden Markov model for detecting postures based on joint points. The postures include standing, lying down, sitting, crouching, squatting, and sitting down. The control unit 101 determines (estimates) the state of the care recipient 80 based on the posture.The control unit 101 can detect a fall, for example, by estimating a standing posture and a lying posture from successive images 1041 (frames) in time series. The control unit 101 can detect a fall, for example, by detecting a lying posture outside the area of ​​the bed 90 after estimating a lying posture within the area of ​​the bed 90. The control unit 101 can detect getting out of bed, for example, by estimating a standing posture outside the area of ​​the bed 90 after estimating a lying posture within the area of ​​the bed 90. The control unit 101 can detect leaving the room, for example, by detecting that a joint point has moved to an area near the door and disappeared there.

[0054] The control unit 101 determines the state of the care recipient 80 based on the image 1041 and the output of the bed sensor 103. When the control unit 101 determines that a predetermined load (for example, a predetermined amount of load) has been detected based on the output of the bed sensor 103, even if the control unit 101 detects specific states, such as a fall, a drop, getting out of bed, and leaving the room, based on the image 1041, it performs control not to notify the detected specific state (i.e., information regarding the state of the monitored person) (hereinafter simply referred to as "not notifying control"). The not notifying control can be control not to send (output) information regarding the state of the monitored person to the server 20 (hereinafter referred to as "first control"). Notification The control not to determine the specific state detected based on the image 1041 as the state of the monitored person may be control (hereinafter referred to as "second control").

[0055] The control of not reporting may be a control (hereinafter referred to as "third control") that does not cause the mobile device 30 to report information about the monitored person's condition even when the mobile device 30 receives the status of the monitored person 80 as information about the monitored person's condition. In the third control, when the control unit 101 transmits the status of the monitored person 80 to the mobile device 30 via the server 20 as information about the monitored person's condition, the control unit 101 may include information that prohibits the monitoring person's condition from being reported (hereinafter referred to as "prohibition information") in the information about the monitored person's condition. In the third control, when the control unit 101 transmits the status of the monitored person 80 to the mobile device 30 via the server 20 as information about the monitored person's condition, the control unit 101 may include information indicating that the monitored person is in a specific state or is undetermined (hereinafter referred to as "undetermined suggestion information") in the information about the monitored person's condition.

[0056] While the bed sensor 103 is detecting a load, it is highly likely that the care recipient 80 is on the bed 90. In this case, even if a specific state is detected as the state of the care recipient 80 based on the image 1041, it is considered that the specific state of the caregiver 70 has actually been erroneously detected as the specific state of the care recipient 80. For this reason, while the bed sensor 103 is detecting a load, the control unit 101 does not transmit (output) the state of the care recipient 80 to the server 20 as information on the state of the monitored person by the first control or the second control, even if the control unit 101 detects a specific state based on the image 1041. This makes it possible to suppress false reports of specific states.

[0057] The control unit 101 calculates the load based on the output of the bed sensor 103, and determines that a predetermined load (e.g., a predetermined amount of load) has been detected when the calculated load exceeds a predetermined threshold set corresponding to the weight of each care recipient 80. The predetermined threshold may be set in advance for each care recipient 80 and stored in the memory of each control unit 101.

[0058] Fig. 5 is an explanatory diagram showing a comparative example in which false alarms of falls occur. Fig. 6 is an explanatory diagram showing an embodiment in which false alarms of falls are suppressed. In these figures, the white arrows indicate the passage of time. In Fig. 6, the bed sensor 103 is colored gray to indicate that a load is being detected by the bed sensor 103.

[0059] In the comparative example of FIG. 5 , the care recipient 80 is asleep on the bed 90. The caregiver 70 enters the room, approaches the bed 90, and provides some kind of care (assistance). In this case, the care recipient 80 is barely moving on the bed 90, and most of his or her body is hidden by the bedding, so there is a possibility that the state of the care recipient 80 will not be detected. Specifically, when the above-described time difference method is used, the object silhouette is not detected, so there is a possibility that the state of the care recipient 80 will not be detected. Furthermore, when the above-described trained model using machine learning is used, there is a possibility that the state of the care recipient 80 will not be detected, because a person region is not detected. In this case, the state of the caregiver 70 is erroneously detected as the state of the care recipient 80 based on the image 1041, and the crouching state of the caregiver 70 while providing care may be erroneously determined as a fall of the care recipient 80.

[0060] In the embodiment of FIG. 6, unlike the comparative example of FIG. 5, the bed sensor 103 further detects the load on the bed 90. Then, since the load is detected by the bed sensor 103, the control unit 101 performs the first control or the second control so as not to notify the detected fall even if it erroneously detects a fall as the state of the care recipient 80 based on the image 1041. As a result, the fall of the care recipient 80 is not transmitted to the server 20. Therefore, false reports of falls are suppressed. Note that when the third control is performed, the fall of the care recipient 80 is transmitted to the mobile terminal 30 via the server 20 together with the prohibition information, and as a result, the fall of the care recipient 80 is not notified on the mobile terminal 30. No notification is given of a fall by the caregiver 80. Similarly, as a result of the fall by the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the undetermined suggestion information, no notification is given of the fall by the care recipient 80 on the mobile terminal 30.

[0061] Fig. 7 is an explanatory diagram showing a comparative example in which false alarms of bed exit occur. Fig. 8 is an explanatory diagram showing an embodiment in which false alarms of bed exit are suppressed. In these figures, the white arrows indicate the passage of time. In Fig. 8, the bed sensor 103 is colored gray to indicate that a load is being detected by the bed sensor 103.

[0062] In the comparative example of FIG. 7 , the care recipient 80 is asleep on the bed 90. Then, the caregiver 70 enters the room, provides some care (assistance) near the bed 90, and then leaves the bed 90. In this case, the care recipient 80 barely moves on the bed 90, and most of his body is hidden by the bedding, so there is a possibility that the state of the care recipient 80 will not be detected, as in the comparative example of FIG. 5 . In this case, the state of the caregiver 70 is erroneously detected as the state of the care recipient 80. That is, the behavior (movement) of the caregiver 70, not the care recipient 80, is tracked (detected). As a result, the behavior of the caregiver 70, who is close to the bed 90, leaving the bed 90 may be erroneously determined as the care recipient 80 getting out of bed.

[0063] In the embodiment of FIG. 8 , unlike the comparative example of FIG. 7 , the bed sensor 103 further detects a load on the bed 90. Since the bed sensor 103 detects a load, the control unit 101 performs the first control or the second control so as not to notify the detected bed getting-out even if the bed getting-out is erroneously detected as the state of the care recipient 80 based on the image 1041. As a result, the bed getting-out of the care recipient 80 is not transmitted to the server 20. Therefore, false bed getting-out notifications are suppressed. When the third control is performed, the bed getting-out of the care recipient 80 is transmitted to the mobile device 30 via the server 20 together with the prohibition information, and therefore the bed getting-out of the care recipient 80 is not notified on the mobile device 30. Similarly, the bed getting-out of the care recipient 80 is transmitted to the mobile device 30 via the server 20 together with the indetermination suggestion information, and therefore the bed getting-out of the care recipient 80 is not notified on the mobile device 30.

[0064] In the notification of bed exit, a predetermined load on the bed 90 can be detected by detecting a predetermined amount of load by the bed sensor 103, but the center of gravity position or distribution may be used instead of detecting a predetermined amount of load. For example, in detecting bed exit, the center of gravity of the load should be detected at the end of the bed 90, but if the center of gravity of the load is detected at the center of the bed, the bed exit may not be notified. Also, the predetermined load may be detected by combining the magnitude of the load and the center of gravity position or distribution.

[0065] Similarly, when informing of a tipping or falling, the position or distribution of the center of gravity may be used instead of the magnitude of the load.

[0066] Fig. 9 is an explanatory diagram showing a comparative example in which false alarms of leaving the room occur. Fig. 10 is an explanatory diagram showing an embodiment in which false alarms of leaving the room are suppressed. In these figures, the white arrows indicate the passage of time. In Fig. 10, the bed sensor 103 is colored gray to indicate that a load is being detected by the bed sensor 103.

[0067] In the comparative example of FIG. 9, the care recipient 80 is asleep on the bed 90. The caregiver 70 provides some care (assistance) near the bed 90 and then leaves the room. In this case, the care recipient 80 is barely moving on the bed 90 and most of his or her body is hidden by the bedding, so there is a possibility that the state of the care recipient 80 will not be detected, as in the comparative example of FIG. 5. In this case, the state of the caregiver 70 will be erroneously detected as the state of the care recipient 80. In other words, the behavior (movement) of the caregiver 70, not the care recipient 80, is tracked (detected). As a result, the action of the caregiver 70, who is close to the bed 90, moving out of the imaging area may be erroneously determined as the care recipient 80 leaving the room. Furthermore, after it is determined that the care recipient 80 has left the room (i.e., the room is vacant), the sensitivity of the status detection based on the image 1041 may be reduced. This is done to prevent false alarms caused by erroneously detecting the status of the care recipient 80 even when the room is vacant. As a method for reducing the sensitivity of the status detection based on the image 1041, for example, when using the time difference method described above, it is possible to reduce the detection sensitivity of the moving silhouette by adjusting the threshold value used in binarization when detecting the moving silhouette. However, if the care recipient 80 is erroneously determined to have left the room (vacant) even when he or she is asleep, a subsequent fall of the care recipient 80 may not be detected because the sensitivity of the status detection based on the image 1041 has been reduced.

[0068] In the embodiment of FIG. 10 , unlike the comparative example of FIG. 9 , the bed sensor 103 further detects a load on the bed 90. Because the bed sensor 103 detects the load, the control unit 101 performs the first control or the second control so as not to notify the detected leaving even if the state of the care recipient 80 is erroneously detected as leaving the room based on the image 1041. As a result, the leaving of the care recipient 80 is not transmitted to the server 20. Therefore, false notifications of getting out of bed are suppressed. Furthermore, in the second control, even if the control unit 101 includes a function to reduce the sensitivity of state detection based on the image 1041 when leaving the room is determined, false detection of leaving the room is suppressed, thereby preventing false notifications of a fall of the care recipient 80. Note that when the third control is performed, the leaving of the care recipient 80 is transmitted to the mobile device 30 via the server 20 together with prohibition information, and as a result, the leaving of the care recipient 80 is not notified on the mobile device 30. Similarly, the leaving of the care recipient 80 is transmitted to the mobile terminal 30 via the server 20 together with the undetermined suggestion information, so that the leaving of the care recipient 80 is not notified on the mobile terminal 30 .

[0069] When the control unit 101 detects a state of falling, dropping, or getting out of bed based on the image 1041, it stores an image relating to that state. When a state of falling, dropping, or getting out of bed is detected based on the image 1041, there is a possibility that (1) the bed sensor 103 has erroneously detected the load on the bed 90, or (2) the load on the bed 90 has been detected because an object has been placed on the bed 90. In this case, by storing an image relating to the state of a fall or the like, it is possible to record the actual fall or the like that has occurred, and also to leave evidence that the caregiver 70 has entered the room or been working.

[0070] When performing the second control, the control unit 101 stores a determination result if it detects any one of a state of a fall, a drop, or getting out of bed based on the image 1041. The determination result may include the following first to third determination results. The control unit 101 may store the first to third determination results separately. The first determination result is, for example, a determination result that the bed sensor 103 detects a load on the bed 90, and a fall or the like is not detected based on the image 1041. The first determination result corresponds to a determination result that a fall or the like has not occurred (for example, sleeping). The second determination result is, for example, a determination result that the bed sensor 103 does not detect a load on the bed 90, and a fall or the like is detected based on the image 1041. The second determination result corresponds to a determination result that a fall or the like has occurred. The third determination result is a determination result that the bed sensor 103 detects a load on the bed 90, and a fall or the like is detected based on the image 1041. The third determination result corresponds to a determination result that a fall or the like is undetermined (there is a possibility of a fall or the like), and is distinguished from the second determination result. This makes it possible to confirm (verify) whether or not a fall or the like has been detected by the image, for example, when a notification of a fall or the like is suppressed due to detection of a load.

[0071] (Server 20) 11 is a block diagram showing the hardware configuration of the server 20. The server 20 includes a control unit 201, a communication unit 202, a storage unit 203, and a monitor 204. They are connected to each other by a bus. The monitor 204 constitutes a notification unit.

[0072] The control unit 201 includes a CPU, RAM, ROM, etc. The control unit 201 controls each unit of the server 20 and performs arithmetic processing according to a program. The functions of the control unit 201 will be described in detail later.

[0073] The communication unit 202 is configured with an interface circuit for communicating with, for example, the mobile terminal 30 via a LAN.

[0074] The storage unit 203 is configured by an HDD or SSD, and stores various programs and various data.

[0075] The monitor 204 displays various information such as the condition of the care recipient 80 in each room.

[0076] The function of the control unit 201 will be described.

[0077] The control unit 201 receives the state of the care recipient 80 from the detection unit 10 via the communication unit 202 as information on the state of the monitored person. The state of the care recipient 80 includes a specific state. The control unit 201 may further receive from the detection unit 10 the respiratory rate, heart rate, etc. calculated based on the output of the bed sensor 103.

[0078] The control unit 201 displays the state of the care recipient 80 received from the detection unit 10 on the monitor 204 .

[0079] 12 is a diagram showing the management table 2031. The control unit 201 stores in advance in the storage unit 203 the management table 2031 in which a unique sensor ID identifying each detection unit 10, an installation location which is the number of the room in which each detection unit 10 is installed, and a person being cared for 80 who is a monitored person residing in the installation location are associated with each other.

[0080] In the example of FIG. 12, for example, a sensor ID of SU-1, a location of the sensor, Room 101, and a care recipient 80, Mr. A, are associated with each other.

[0081] The control unit 201 refers to the management table 2031 and identifies the location of the detection unit 10 and the person being cared for 80 being monitored, based on the sensor ID of the detection unit 10 that transmitted the status of the person being cared for 80. The control unit 201 can display the status of each person being cared for 80 on the monitor screen of the monitor 204 together with the location of the detection unit 10.

[0082] The control unit 201 detects the occurrence of an event based on the state of the care recipient 80 received from the detection unit 10. An event is a change in the state of the care recipient 80, which requires care for the care recipient 80 and for which an alert (notification) should be issued to the caregiver 70. Events may include specific states such as falling, tripping, getting out of bed, and leaving the bed. The control unit 201 may detect abnormal breathing and abnormal heart rate as the occurrence of an event based on the respiratory rate and heart rate received from the detection unit 10.

[0083] The control unit 201 manages the addresses of the mobile terminals 30, and may send an event notification to the mobile terminals 30 at the managed addresses to notify them of a detected event. The event notification includes information about the state of the monitored person. If the control unit 201 does not manage the addresses of the mobile terminals 30, it may send the event notification without specifying a destination. Note that a port number for each mobile terminal 30 to access the server 20 and obtain an event notification may be set in advance in the server 20. In this case, each mobile terminal 30 may receive an event notification from the server 20. By inputting the IP address of the mobile terminal 30 and the set port number, the mobile terminal 30 can access the server 20 and obtain event information from the server 20.

[0084] The control unit 201 may execute some or all of the functions of the control unit 101 of the detection unit 10 on behalf of the control unit 101. In this case, the control unit 201 may receive the image 1041 and the output of the bed sensor 103 from the detection unit 10. The control unit 201 may determine the state of the care recipient 80 based on the image 1041 and the output of the bed sensor 103.

[0085] (Mobile terminal 30) 13 is a block diagram showing the hardware configuration of mobile terminal 30. Mobile terminal 30 has a control unit 301, a wireless communication unit 302, a storage unit 303, an input / display unit 304, and an audio input / output unit 305, which are interconnected by a bus. Mobile terminal 30 may be configured by a portable communication terminal device such as a tablet computer, a smartphone, or a mobile phone.

[0086] The control unit 301 includes a CPU, RAM, ROM, etc. The control unit 301 controls each unit of the mobile terminal 30 and performs arithmetic processing in accordance with a program. The functions of the control unit 301 will be described in detail later.

[0087] The wireless communication unit 302 communicates wirelessly with each device directly or via the wireless AP 31 using wireless communication standards such as Wi-Fi and Bluetooth (registered trademark). As described above, the mobile terminal 30 may be configured to be able to access the server 20 by inputting connection information with the server 20 (the IP address of the server 20 and the port number set on the server 20) in addition to the wireless LAN settings. The connection information is information for identifying the server 20 and may be stored in the storage unit 303.

[0088] The storage unit 303 is configured by an SSD or an SD card, and stores various programs and various data.

[0089] The input display unit 304 is configured, for example, by a touch panel in which a touch sensor is superimposed on a display surface such as a liquid crystal display. The input display unit 304 displays a terminal monitor screen corresponding to the monitor screen displayed on the monitor 204 of the server 20.

[0090] The voice input / output unit 305 is, for example, a speaker and a microphone, and enables the caregiver 70 to make voice calls to other mobile terminals 30 via the wireless communication unit 302 .

[0091] The function of the control unit 301 will be described.

[0092] The control unit 301 continuously receives event notifications from the server 20. The control unit 301 continuously receives the status of each care recipient 80 from the server 20 as information relating to the status of the monitored person, and displays it on the terminal monitor screen. As described above, the event notification includes information relating to the status of the monitored person.

[0093] When the condition of each care recipient 80 is in a specific state, the control unit 301 can notify the condition of each care recipient 80 as information on the condition of the person being monitored by displaying on the terminal monitor screen, sound, vibration, light emission, or a combination of these. The sound, vibration, and light emission functions can be functions that are standard features of the mobile terminal 30.

[0094] When the information relating to the condition of the person being monitored includes prohibition information or uncertain suggestion information, the control unit 301 does not report the condition of the person being cared for 80 as information relating to the condition of the person being monitored.

[0095] The operation of the monitoring system 1 will now be described.

[0096] 14 is a flowchart showing the operation of the monitoring system 1. This flowchart can be executed by the control unit 101 of the detection unit 10 in accordance with a program. Note that when the server 20 substitutes for some or all of the functions of the control unit 101 of the detection unit 10, this flowchart can be executed by the control unit 201 of the server 20 in accordance with a program.

[0097] The control unit 101 acquires the image 1041 from the camera 104 by receiving it (S101), or the like. The control unit 101 acquires the output of the bed sensor 103 by receiving it (S102).

[0098] The control unit 101 detects the state of the care recipient 80 based on the image 1041 (S103).

[0099] The control unit 101 determines whether the state of the care recipient 80 is a specific state (S104). If the control unit 101 determines that the state of the care recipient 80 is not a specific state (S104: NO), it determines the state of the care recipient 80 detected in step S103 as the state of the care recipient 80 (S106).

[0100] When the control unit 101 determines that the state of the care recipient 80 is a specific state (S104: YES), the control unit 101 determines whether or not a load is detected (S105) based on the output of the bed sensor 103. When the control unit 101 determines that a load is detected (S105: YES), it executes control not to issue a notification (S107).

[0101] When the control unit 101 determines that no load is detected (S105: NO), it determines the specific state of the care recipient 80 detected in step S103 as the state of the care recipient 80 (S106).

[0102] This embodiment has the following advantages.

[0103] The system determines the condition of the person being monitored based on an image of an area including at least the area around the bed and the output of a sensor that detects the load on the bed, and reports information about the monitored person's condition based on the determination. When a predetermined load is detected by the sensor, the system controls not to report information about the monitored person's condition even if it detects a fall, a slip, or getting out of bed based on the image. This makes it possible to prevent false reports of the monitored person falling, etc.

[0104] Furthermore, the control not to notify is set to a control not to output the information, which makes it possible to suppress false alarms of the monitored person's tumbling or falling with a simple configuration.

[0105] In addition, the control not to report is set to a control not to report information even if the terminal that reports information about the monitored person's condition receives the information, which makes it easy to prevent false reports of the monitored person falling or tripping.

[0106] Furthermore, the control not to issue a notification is set to a control not to determine the state detected based on the image as the state of the monitored person, which makes it possible to flexibly suppress false alarms such as the monitored person falling or tripping over with a simple configuration.

[0107] Furthermore, if a state of falling, dropping, or getting out of bed is detected based on the image, the image relating to the state of the monitored person is stored. By storing images of the condition, it is possible to record actual falls and other incidents, as well as to leave evidence that caregivers have entered the room and are working.

[0108] Furthermore, if a fall, drop, or getting out of bed is detected based on the image, the detection result is stored. This makes it possible to confirm whether a fall or other condition was detected based on the image, for example, when a load is detected.

[0109] Furthermore, the specific state includes leaving the room, which can reduce false alarms of leaving.

[0110] In addition, the control not to report is set to a control in which information prohibiting reporting is included in the information about the monitored person's condition and transmitted to the terminal that reports the information about the monitored person's condition, which makes it easier to prevent false reports of the monitored person falling or tripping.

[0111] Furthermore, the system calculates the pixel-by-pixel difference between adjacent frames of the image and binarizes the difference to detect a moving silhouette, based on which the state of the monitored person can be determined. This makes it possible to detect the state of the monitored person easily and with high accuracy.

[0112] Furthermore, a trained model based on machine learning is used to estimate joint points from images, and the state of the monitored person is determined based on the estimated joint points. This makes it possible to detect the state of the monitored person easily and with higher accuracy.

[0113] Furthermore, when it is determined that the person has left the room, the threshold value for the binarization is adjusted to lower the detection sensitivity of the moving silhouette, thereby reducing false alarms when the room is vacant.

[0114] Furthermore, the imaging area is set to include the area on the bed and the area around the bed, which further reduces false alarms due to the monitored person falling or tripping.

[0115] The configuration of the monitoring system 1 described above is a description of the main configuration in explaining the features of the above-described embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims. For example, the above-described flowchart may include steps other than those shown in the flowchart, or some steps may not be included. The order of the steps is also not limited to the above-described embodiment. Furthermore, each step may be combined with other steps and executed as a single step, may be included in other steps and executed, or may be divided into multiple steps and executed.

[0116] The means and methods for performing various processes in the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The above-described program can be stored on a USB memory or a DVD (Digital Versatile provided by a computer-readable recording medium such as a Serial Disc (Serial No. 0 ... The program may be provided in a computer-readable storage medium, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable storage medium is usually transferred to and stored in a storage device such as a hard disk. The program may be provided as standalone application software, or may be incorporated as a function into the software of the device. [Explanation of symbols]

[0117] 1. Monitoring system, 10 detection unit, 101 control section, 102 storage section, 103 Bed Sensor, 104 cameras, 1041 images, 20 servers, 201 control section, 202 Communications Department, 203 Memory section, 204 monitors, 30 mobile devices, 301 control section, 302 Radio Communications Department, 303 Memory section, 304 input display unit, 305 audio input / output unit, 70 Caregiver, 80 Caregiver, 90 beds, 91 Area surrounding the bed.

Claims

1. an imaging unit that captures an imaging area including at least the area surrounding the bed; A sensor that detects the load on the bed; a determination unit that determines the state of the monitored person based on the captured image and the output of the sensor; a terminal that notifies information about the state of the monitored person based on the determination by the determination unit, When it is determined that a specified load has been detected based on the output of the sensor, the monitoring system performs control not to report the information even if it detects any of the following conditions based on the image: falling, tripping, or getting out of bed.

2. The monitoring system according to claim 1 , wherein the control not to notify is performed by the determination unit not outputting the information.

3. The monitoring system according to claim 1 , wherein the control not to report is to not report the information even if the terminal receives the information.

4. The monitoring system according to claim 1 , wherein the control of not reporting is such that the determining unit does not determine the state detected based on the image as the state of the monitored person.

5. an image storage unit that stores the image captured by the imaging unit; The monitoring system according to claim 2 or 3, wherein when any one of the states of falling, slipping, and getting out of bed is detected based on the image, an image relating to the state is stored in the image storage unit.

6. a determination result storage unit that stores the determination result of the determination unit; The monitoring system according to claim 2 , wherein when any one of the states of falling, slipping off, and getting out of bed is detected based on the image, the determination result is stored in the determination result storage unit.

7. The monitoring system according to claim 4 , further including, as the state, leaving the room.

8. The monitoring system according to claim 3 , wherein the determination unit transmits the information to the terminal by including information prohibiting notification in the information.

9. The monitoring system according to any one of claims 1 to 8, wherein the determination unit calculates the pixel-by-pixel difference between adjacent frames of the image and determines the state of the monitored person based on a moving silhouette detected by binarizing the difference.

10. The monitoring system according to any one of claims 1 to 8, wherein the determination unit estimates joint points from the image using a trained model based on machine learning, and determines the state of the monitored person based on the estimated joint points.

11. The monitoring system according to claim 9 , which cites claim 7 , wherein the determination unit, when determining that the person has left the room, adjusts a threshold value for the binarization to reduce detection sensitivity of the moving object silhouette.

12. The monitoring system according to any one of claims 1 to 11, wherein the photographing area is an area including the area on the bed and the area around the bed.

13. The image includes at least the area around the bed, and the load on the bed is detected. an acquisition unit that acquires the output of the sensor that outputs the signal; a determination unit that determines the state of the monitored person based on the image and the output of the sensor; a notification unit that notifies information about the state of the monitored person based on the determination by the determination unit, When it is determined that a specified load has been detected based on the output of the sensor, the monitoring device performs control not to report the information even if it detects any of the following conditions based on the image: falling, tripping, or getting out of bed.

14. The monitoring device according to claim 13 , wherein the control not to notify is performed by the determination unit not outputting the information.

15. The monitoring device according to claim 13 , wherein the control not to notify is such that the determining unit does not determine the state detected based on the image as the state of the monitored person.

16. an image storage unit for storing the captured image; The monitoring device according to claim 14 , wherein when the state of falling, dropping, or getting out of bed is detected based on the image, an image relating to the state is stored in the image storage unit.

17. a determination result storage unit that stores the determination result of the determination unit; The monitoring device according to claim 14 or 16, wherein when any one of the states of falling, slipping off, and getting out of bed is detected based on the image, the determination result is stored in the determination result storage unit.

18. The monitoring device according to claim 15 , further including, as the state, leaving the room.

19. The monitoring device described in claim 13, wherein the control not to notify is performed by the judgment unit including information prohibiting notification in the information and transmitting information regarding the monitored person's status based on the judgment by the judgment unit to a terminal that will notify.

20. A step (a) of acquiring an image of an imaging area including at least a peripheral area of ​​the bed and an output of a sensor that detects a load on the bed; (b) determining the state of the monitored person based on the image and the output of the sensor; a step (c) of notifying information regarding the state of the monitored person based on the determination in the step (b); A monitoring method comprising: when it is determined that a specified load has been detected based on the output of the sensor, a step (d) of performing control not to report the information in step (c) even if any of the states of falling, tripping, or getting out of bed is detected based on the image.

21. The monitoring method according to claim 20 , wherein the control not to notify is to not output the information in the step (b).

22. The monitoring method according to claim 20, wherein the control not to notify is to not determine in step (b) the state detected based on the image as the state of the monitored person.

23. 22. The monitoring method according to claim 21, further comprising the step (e) of storing an image relating to a state of falling, dropping, or getting out of bed, when the state of falling is detected based on the image.

24. When any of the states of falling, dropping, and getting out of bed is detected based on the image, The monitoring method according to claim 21 or 23, further comprising a step (f) of storing the determination result in the step (b).

25. The monitoring method according to claim 22 , further including, as the state, leaving the room.

26. The monitoring method described in claim 20, wherein the control not to notify is to include information prohibiting notification in the information in step (b) and send information regarding the monitored person's status based on the judgment to a terminal that notifies.

27. A monitoring program for causing a computer to execute the monitoring method according to any one of claims 20 to 26.

Images