Monitoring system, monitoring device, monitoring method, and monitoring program

Abstract

To provide a watching system, a watching device, a watching method, and a watching program capable of suppressing an erroneous report of the overturn, fall or the like of a monitored person.SOLUTION: In a watching system in which a plurality of detection sections, a server, and portable terminals are mutually communicably connected via a LAN (Local Area Network) and a wireless AP, the detection section 10 has: an imaging section (camera 104) for photographing a photographing area including at least a peripheral area of a bed; a sensor (bed sensor 103) for detecting a load on the bed; a determination section (control section 101) for determining the state of a monitored person on the basis of an image and the output of the sensor; and a terminal for reporting information related to the state of the monitored person based on the determination. When it is determined that a predetermined load is detected on the basis of the output of the sensor, control is performed not to report the information related to the state of the monitored person even when any one of the overturn, fall and bed-leaving is detected on the basis of the image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a monitoring system, a monitoring device, a monitoring method, and a monitoring program.

Background Art

[0002] In care facilities for the elderly and hospitals, etc., care recipients (hereinafter referred to as "care recipients") who are living or hospitalized are at risk of falling while walking or falling from a bed and getting injured within the care facility. Therefore, in order for nurses and care workers (hereinafter referred to as "caregivers") to rush to the care recipient immediately when they are in such a state, the development of a system for constantly detecting the state of the care recipient is underway.

[0003] The following Patent Document 1 discloses the following prior art. The peripheral area of the bed is photographed from above at a position higher than the height of the person being monitored above the floor surface to obtain image data, skeleton information is extracted from the image data, and the posture of the person being monitored is detected based on the skeleton information and the skeleton information sample stored in advance. Then, based on the detected posture, it is determined whether the person being monitored has fallen or tumbled.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, based on the finding in the prior art disclosed in Patent Document 1 that falls occur on the floor other than the bed, the presence or absence of a monitored person's fall is determined based on image data with the peripheral area of the bed as the imaging area. Therefore, when a caregiver is working beside the bed to take care of a monitored person lying in the bed and makes a crouching motion or the like, it may be determined that the monitored person has fallen.

[0006] Also, even if the imaging area is set to an area including the bed, since the monitored person on the bed is covered with a futon, it is difficult to recognize the monitored person as a human from the image, and it becomes even more difficult to recognize when the monitored person shows no movement. Therefore, even if the area including the bed is set, it is impossible to prevent false alarms such as falls due to misidentifying the posture of a caregiver working beside the bed as the posture of the monitored person.

[0007] The present invention has been made to solve such problems. That is, an object of the present invention is to provide a monitoring system, a monitoring device, a monitoring method, and a monitoring program capable of suppressing false alarms such as falls of a monitored person.

Means for Solving the Problems

[0008] The above problems of the present invention are solved by the following means.

[0009] (1) An imaging unit that images an imaging area including at least the peripheral area of the bed, a sensor that detects the load on the bed, a determination unit that determines the state of the monitored person based on the captured image and the output of the sensor, and a terminal that notifies information regarding the state of the monitored person based on the determination by the determination unit. When it is determined based on the output of the sensor that a predetermined load is detected, control is performed not to notify the information even if any of the states of fall, tumble, or getting out of bed is detected based on the image. A monitoring system.

[0010] (2) The non - notification control in the above - mentioned (1) monitoring system is that the determination unit does not output the information.

[0011] (3) The non - notification control in the above - mentioned (1) monitoring system is that even if the terminal receives the information, it does not notify.

[0012] (4) The non - notification control in the above - mentioned (1) monitoring system is that the determination unit does not determine the state detected based on the image as the state of the person being monitored.

[0013] (5) The monitoring system according to the above - mentioned (2) or (3) has an image storage unit that stores the image captured by the imaging unit, and when any of the states of falling, tumbling, or getting out of bed is detected based on the image, the image related to the state is stored in the image storage unit.

[0014] (6) The monitoring system according to any of the above - mentioned (2), (3), and (5) has a determination result storage unit that stores the determination result of the determination unit, and when any of the states of falling, tumbling, or getting out of bed is detected based on the image, it is stored in the determination result storage unit.

[0015] (7) In the monitoring system according to the above - mentioned (4), the state further includes leaving the room.

[0016] C (8) In the monitoring system according to the above - mentioned (3), the determination unit includes information for prohibiting notification in the information and transmits it to the terminal.

[0017] (9) In the monitoring system according to any of the above - mentioned (1) to (8), the determination unit calculates the difference between pixels for each adjacent frame of the image, and determines the state of the person being monitored based on the moving object silhouette detected by binarizing the difference.

[0018] (10) The determination unit uses a learned model by machine learning to estimate joint points from the image, and determines the state of the monitored person based on the estimated joint points. The monitoring system according to any one of (1) to (8) above.

[0019] (11) When the determination unit determines the leaving of the room, it adjusts the threshold value in the binarization to reduce the detection sensitivity of the moving object silhouette. The monitoring system according to (9) above, which quotes (7).

[0020] (12) The imaging area is an area including the area on the bed and the peripheral area of the bed. The monitoring system according to any one of (1) to (11) above.

[0021] (13) An acquisition unit that acquires an image of an imaging area including at least the peripheral area of the bed and an output of a sensor that detects the load on the bed, and a determination unit that determines the state of the monitored person based on the image and the output of the sensor. And a notification unit that notifies information regarding the state of the monitored person based on the determination by the determination unit. When it is determined that a predetermined load is detected based on the output of the sensor, even if any of the states of falling, tumbling, or getting out of bed is detected based on the image, control is performed not to notify the information. Monitoring device.

[0022] (14) The control not to notify is that the determination unit does not output the information. The monitoring device according to (13) above.

[0023] (15) The control not to notify is that the determination unit does not determine the state detected based on the image as the state of the monitored person. The monitoring device according to (13) above.

[0024] (16) It has an image storage unit that stores the captured image. When any of the states of falling, tumbling, or getting out of bed is detected based on the image, an image related to the state is stored in the image storage unit. The monitoring device according to (14) above.

[0025] (17) It has a determination result storage unit that stores the determination result of the determination unit, and when any one of the states of falling, tumbling, and getting out of bed is detected based on the image, it stores in the determination result storage unit. The monitoring device according to (14) or (16) above.

[0026] (18) The monitoring device according to (15) above, wherein the state further includes leaving the room.

[0027] (19) The non - notification control is that the determination unit includes information for prohibiting notification in the information, and transmits information regarding the state of the monitored person based on the determination by the determination unit to a terminal that notifies the information. The monitoring device according to (13) above.

[0028] (20) Step (a) of acquiring an image of a shooting area including at least the peripheral area of the bed and an output of a sensor that detects the load on the bed; step (b) of determining the state of the monitored person based on the image and the output of the sensor; step (c) of notifying information regarding the state of the monitored person based on the determination in step (b); and step (d) of, when it is determined based on the output of the sensor that a predetermined load is detected, performing control not to notify the information in step (c) even if any one of the states of falling, tumbling, and getting out of bed is detected based on the image. A monitoring method having these steps.

[0029] (21) The non - notification control is not outputting the information in step (b). The monitoring method according to (20) above.

[0030] (22) The non - notification control is not determining the state detected based on the image in step (b) as the state of the monitored person in step (b). The monitoring method according to (20) above.

[0031] (23) When any one of the states of falling, tumbling, and getting out of bed is detected based on the image, it has step (e) of storing an image regarding the state. The monitoring method according to (21) above.

[0032] (24) When any one of the states of falling, tumbling, or getting out of bed is detected based on the image, the monitoring method according to (21) or (23) above includes a step (f) of storing the determination result in step (b).

[0033] (25) The monitoring method according to (22) above, wherein the state further includes leaving the room.

[0034] (26) The control of not notifying is to include information for prohibiting notification in the information in step (b) and transmit it to a terminal that notifies information regarding the state of the monitored person based on the determination. The monitoring method according to (20) above.

[0035] (27) A monitoring program for causing a computer to execute the monitoring method according to any one of (20) to (26) above.

Effect of the Invention

[0036] Based on an image of an area including at least the peripheral area of the bed and the output of a sensor that detects the load on the bed, the state of the monitored person is determined, information regarding the state of the monitored person based on the determination is notified, and when a predetermined load is detected by the sensor, even if any one of falling, tumbling, or getting out of bed is detected based on the image, control is performed not to notify information regarding the state of the monitored person. Thereby, false notifications such as the monitored person falling or tumbling can be suppressed.

Brief Description of the Drawings

[0037]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0038] Hereinafter, with reference to the drawings, a monitoring device, a monitoring system, a monitoring method, and a monitoring program according to an embodiment of the present invention will be described. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0039] FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 according to the present embodiment. The monitoring system 1 used in a nursing care facility or the like includes a detection unit 10, a server 20, a plurality of mobile terminals 30, and a wireless AP (access point) 31. The detection unit 10, the server 20, the mobile terminal 30, and the detection unit 10 are connected to be communicable with each other via a LAN (Local Area Network) and the wireless AP 31. The mobile terminal 30 is carried by a caregiver 70 who nurses or cares for the care recipient 80. The detection unit 10 constitutes a monitoring device.

[0040] (Detection Unit 10) FIG. 2 is a diagram showing the interior of the room of the care recipient 80 (hereinafter also referred to as the "living room") where the detection unit 10 is arranged. FIG. 3 is a block diagram showing the hardware configuration of the detection unit 10. As shown in FIG. 3, the detection unit 10 includes a bed sensor 103 and a camera 104. The bed sensor 103 constitutes a sensor. The camera 104 constitutes an imaging unit.

[0041] As shown in FIG. 2, the bed sensors 103 are respectively arranged on the bed 90 in the living room of the care recipient 80 who is the person to be monitored. Specifically, the bed sensors 103 are arranged so as to be laid under the mattress on the bed 90, for example. The camera 104 can be arranged on the ceiling or the upper part of the wall of the living room, or attached to the bed 90 (see FIG. 6 etc.).

[0042] As shown in FIG. 3, the detection unit 10 includes a control unit 101, a communication unit 102, a bed sensor 103, and a camera 104, which are interconnected by a bus. The control unit 101 constitutes a determination unit, an image storage unit, and a determination result storage unit. The control unit 101, together with the communication unit 102, constitutes an acquisition unit and a notification unit.

[0043] The control unit 101 includes a CPU, a RAM, a ROM, etc. The control unit 101 may include an HDD or an SSD. The control unit 101 performs control and arithmetic processing of each part of the detection unit 10 according to a program. Details of the functions of the control unit 101 will be described later.

[0044] The communication unit 102 is an interface circuit (for example, a LAN card, a wireless communication circuit, etc.) for communicating with, for example, the server 20 etc. via a LAN.

[0045] The bed sensor 103 outputs a signal corresponding to the load on the bed 90. The bed sensor 103 can be constituted by, for example, a sheet-like bag filled with air and a piezoelectric sensor for detecting fluctuations in the pressure of this air.

[0046] The camera 104 images a shooting area including at least the peripheral area 91 (see FIG. 4) of the bed 90, and outputs the captured image 1041 (image data). Hereinafter, the image 1041 output by the camera 104 is also simply referred to as "image 1041". The shooting area may be an area including above the bed 90 (including a part above the bed 90) and the peripheral area 91 of the bed 90. The shooting area may be an area including only the peripheral area 91 of the bed 90.

[0047] The image 1041 includes still images and moving images. The camera 104 is a near-infrared camera, but a visible light camera may be used instead, or these may be used in combination.

[0048] FIG. 4 is a diagram showing the image 1041. The image 1041 shown in FIG. 4 is an image 1041 captured by the camera 104 attached to the bed 90. FIG. 4 shows an image 1041 in which an area including above the bed 900 and the peripheral area 91 of the bed 90 is imaged as the shooting area.

[0049] The functions of the control unit 101 will be described.

[0050] The control unit 101 detects the body movement of the cared-for person 80 on the bed 90 based on the output of the bed sensor 103. Thereby, the control unit 101 can detect the respiration rate, heart rate, etc. of the cared-for person 80, and transmit the detection results to the server 20. The control unit 101 may further detect sleep, waking up, etc. based on the output of the bed sensor 103, and transmit the detection results to the server 20.

[0051] The control unit 101 determines the state of the cared-for person 80 based on the image 1041 and the output of the bed sensor 103, and transmits (outputs) it to the server 20 as information regarding the state of the person being monitored. The state of the cared-for person 80 includes falls, drops, getting out of bed, and leaving the room, which are specific states. The state of the cared-for person 80 includes states other than specific states, such as walking and washing the face.

[0052] The control unit 101 can detect the state of the care recipient 80 by, for example, either of the following two methods. As a first method, the control unit 101 calculates the difference between pixels for each frame of the image 1041 and binarizes the difference to detect a moving object silhouette (temporal difference method). Then, the control unit 101 detects the state of the care recipient 80 based on the moving object silhouette. Specifically, the control unit 101 can detect a fall, for example, based on the fact that the center of gravity of the detected moving object silhouette has suddenly changed from a state of moving in a time series to a state of stopping, and a change in the aspect ratio of the rectangle corresponding to the moving object silhouette. The control unit 101 can detect a fall, for example, based on the fact that the moving object silhouette has suddenly changed from a state of existing within the area of the bed 90 to a state of existing outside the area of the bed 90, and the overlapping width between the moving object silhouette and the boundary of the area of the bed 90. The area of the bed 90 in the image 1041 is preset when the detection unit 10 is installed in each room and can be stored in the memory of the control unit 101 as data. The control unit 101 can detect getting out of bed, for example, based on the fact that the moving object silhouette has moved from a state of existing within the area of the bed 90 to outside the area of the bed 90. The control unit 101 can detect leaving the room, for example, based on the fact that the moving object silhouette has moved to an area near the door and disappeared in the area near the door. The area near the door in the image 1041 can be stored in the memory of the control unit 101 as data, similar to the area of the bed 90.

[0053] As a second method, the control unit 101 can estimate joint points from the image 1041 using a trained model by machine learning and detect the state of the care recipient 80 based on the estimated joint points. Specifically, for each frame of the image 1041, the control unit 101 detects the region where an object exists, and estimates the category of the object included in the detected region, thereby detecting the person region. The region where an object exists can be detected as a candidate rectangle, which is a rectangle containing the object on the image 1041. The control unit 101 detects the person region by detecting the candidate rectangle among the detected candidate rectangles for which the category of the object is estimated to be a person. The person region can be detected using a neural network. Examples of the method for detecting the person region 610 by a neural network include known methods such as Faster R-CNN, Fast R-CNN, and R-CNN. The neural network for detecting the person region from the image 1041 is pre-trained to detect (estimate) the person region from the image 1041 using the training data of the combination of the image 1041 and the person region set as the correct answer for the image 1041. The control unit 101 detects joint points from the person region using a known technique using a neural network such as DeepPose. The joint points can be detected as coordinates in the image 1041. The neural network for detecting joint points from the person region is pre-trained to identify (estimate) joint points from the person region using the training data of the combination of the person region and the joint points set as the correct answer for the person region. The control unit 101 estimates the posture of the care recipient 80 based on the joint points. The posture can be estimated using a neural network for detecting the posture from the joint points. In this case, the neural network for detecting the posture from the joint points is pre-trained to estimate the posture from the joint points using the training data of the combination of the joint points and the posture set as the correct answer for the joint points. The posture may be estimated using a hidden Markov model for detecting the posture based on the joint points. The posture includes standing position, lying position, sitting position, half-sitting position, squatting, and slumping, etc. The control unit 101 determines (estimates) the state of the care recipient 80 based on the posture.The control unit 101 can detect a fall, for example, when the standing posture and the lying posture are respectively estimated from a series of consecutive images 1041 (frames). The control unit 101 can detect a fall out of the bed, for example, after the lying posture within the area of the bed 90 is estimated and then the lying posture outside the area of the bed 90 is detected. The control unit 101 can detect getting out of bed, for example, after the lying posture within the area of the bed 90 is estimated and then the standing posture outside the area of the bed 90 is detected. The control unit 101 can detect leaving the room, for example, when the joint points move to the area near the door and disappear in the area near the door.

[0054] The control unit 101 determines the state of the care recipient 80 based on the image 1041 and the output of the bed sensor 103. When the control unit 101 determines that a predetermined load (for example, a predetermined amount of load) is detected based on the output of the bed sensor 103, even if it detects a specific state such as a fall, a fall out of the bed, getting out of bed, or leaving the room based on the image 1041, it performs control not to notify the detected specific state (that is, information regarding the state of the person being monitored) (hereinafter simply referred to as "non - notification control"). The non - notification control may be control not to transmit (output) information regarding the state of the person being monitored to the server 20 (hereinafter referred to as "first control"). The non - notification control may also be control not to determine the specific state detected based on the image 1041 as the state of the person being monitored (hereinafter referred to as "second control").

[0055] The control of not notifying means that even if the mobile terminal 30 receives the state of the care recipient 80 as information regarding the state of the person under surveillance, it may be the control of not notifying the mobile terminal 30 of the information regarding the state of the person under surveillance (hereinafter referred to as "the third control"). In the third control, when the control unit 101 transmits the state of the care recipient 80 to the mobile terminal 30 via the server 20 as information regarding the state of the person under surveillance, the control unit 101 may transmit the information including information (hereinafter referred to as "prohibited information") for prohibiting the notification of the information regarding the state of the person under surveillance. Also, in the third control, when the control unit 101 transmits the state of the care recipient 80 to the mobile terminal 30 via the server 20 as information regarding the state of the person under surveillance, the control unit 101 may transmit the information including information (hereinafter referred to as "uncertainty suggestion information") indicating that the person under surveillance is in a specific state or is uncertain.

[0056] While the load is detected by the bed sensor 103, it is highly likely that the care recipient 80 is on the bed 90. In this case, even if a specific state is detected as the state of the care recipient 80 based on the image 1041, it is considered that the specific state of the caregiver 70 is erroneously detected as the specific state of the care recipient 80. For this reason, while the load is detected by the bed sensor 103, even if a specific state is detected based on the image 1041, the control unit 101 does not transmit (output) the state of the care recipient 80 to the server 20 as information regarding the state of the person under surveillance by the first control or the second control. Thereby, false alarms of the specific state can be suppressed.

[0057] The control unit 101 calculates the load based on the output of the bed sensor 103, and determines that a predetermined load (for example, a predetermined amount of load) is detected when the calculated load exceeds a predetermined threshold set corresponding to the weight of each care recipient 80. The predetermined threshold is set in advance for each care recipient 80 and may be stored in the memory of each control unit 101.

[0058] FIG. 5 is an explanatory diagram showing a comparative example in which a false alarm of a fall occurs. FIG. 6 is an explanatory diagram showing an embodiment in which a false alarm of a fall is suppressed. In these figures, the white arrows indicate the passage of time. In FIG. 6, the fact that a load is detected by the bed sensor 103 is indicated by the bed sensor 103 being colored gray.

[0059] In the comparative example of FIG. 5, the care recipient 80 is in a sleeping state on the bed 90. Then, the caregiver 70 enters the room, approaches the bed 90, and performs some care (assistance). In this case, since the care recipient 80 hardly moves on the bed 90 and most of the body is hidden by the futon, there is a possibility that the state of the care recipient 80 may not be detected. Specifically, when using the time difference method described above, there is a possibility that the state of the care recipient 80 may not be detected because the object silhouette is not detected. Also, when using the learned model by machine learning described above, there is a possibility that the state of the care recipient 80 may not be detected because the person area is not detected. In this case, based on the image 1041, the state of the caregiver 70 is erroneously detected as the state of the care recipient 80, so that the crouched state when the caregiver 70 provides care may be erroneously determined as a fall of the care recipient 80.

[0060] In the embodiment of FIG. 6, different from the comparative example of FIG. 5, the load on the bed 90 is further detected by the bed sensor 103. Then, since the load is detected by the bed sensor 103, the control unit 101 performs the first control or the second control so as not to notify the detected fall even if a fall of the care recipient 80 is erroneously detected based on the image 1041. As a result, the fall of the care recipient 80 is not transmitted to the server 20. Therefore, a false alarm of a fall is suppressed. When the third control is performed, as a result of the fall of the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the prohibited information, the fall of the care recipient 80 is not notified on the mobile terminal 30. Similarly, as a result of the fall of the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the undetermined suggestion information, the fall of the care recipient 80 is not notified on the mobile terminal 30.

[0061] FIG. 7 is an explanatory diagram showing a comparative example in which a false alarm of getting out of bed occurs. FIG. 8 is an explanatory diagram showing an embodiment in which a false alarm of getting out of bed is suppressed. In these figures, the white arrows indicate the passage of time. In FIG. 8, the fact that a load is detected by the bed sensor 103 is indicated by the bed sensor 103 being colored gray.

[0062] In the comparative example of FIG. 7, the care recipient 80 is in a sleeping state on the bed 90. Then, the caregiver 70 enters the room, performs some care (assistance) near the bed 90, and then leaves the bed 90. In this case, since the care recipient 80 hardly moves on the bed 90 and most of the body is hidden by the futon, as in the comparative example of FIG. 5, the state of the care recipient 80 may not be detected. In this case, the state of the caregiver 70 is erroneously detected as the state of the care recipient 80. That is, instead of the care recipient 80, the action (movement) of the caregiver 70 is tracked (detected). As a result, the action of the caregiver 70 leaving the vicinity of the bed 90 can be erroneously determined as the care recipient 80 getting out of bed.

[0063] In the embodiment of FIG. 8, different from the comparative example of FIG. 7, the load on the bed 90 is further detected by the bed sensor 103. Then, since a load is detected by the bed sensor 103, the control unit 101 performs the first control or the second control so as not to notify the detected getting out of bed even if the getting out of bed of the care recipient 80 is erroneously detected based on the image 1041. As a result, the getting out of bed of the care recipient 80 is not transmitted to the server 20. Therefore, a false alarm of getting out of bed is suppressed. When the third control is performed, as a result of the getting out of bed of the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the prohibition information, the getting out of bed of the care recipient 80 is not notified on the mobile terminal 30. Similarly, as a result of the getting out of bed of the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the undetermined suggestion information, the getting out of bed of the care recipient 80 is not notified on the mobile terminal 30.

[0064] In the notification of getting out of bed, a predetermined load on the bed 90 can be detected by detecting a predetermined amount of load by the bed sensor 103. Instead of detecting a predetermined amount of load, the center of gravity position or distribution may be used. For example, in the detection of getting out of bed, the center of gravity of the load should be detected at the end of the bed 90. However, when the center of gravity of the load is detected at the center of the bed, the notification of getting out of bed may not be given. Also, a predetermined load may be detected by a combination of the magnitude of the load and the center of gravity position or distribution.

[0065] Note that in the notification of falling down or toppling over, similarly, instead of the magnitude of the load, the center of gravity position or distribution may be used.

[0066] FIG. 9 is an explanatory diagram showing a comparative example in which a false alarm of leaving the room occurs. FIG. 10 is an explanatory diagram showing an embodiment in which a false alarm of leaving the room is suppressed. In these figures, the white arrows indicate the passage of time. In FIG. 10, the fact that a load is detected by the bed sensor 103 is shown by coloring the bed sensor 103 gray.

[0067] In the comparative example of FIG. 9, the care recipient 80 is in a sleeping state on the bed 90. Then, after the caregiver 70 has provided some care (assistance) near the bed 90, the caregiver leaves the room. In this case, since the care recipient 80 hardly moves on the bed 90 and most of the body is hidden by the futon, as in the comparative example of FIG. 5, there is a possibility that the state of the care recipient 80 may not be detected. In this case, the state of the caregiver 70 may be erroneously detected as the state of the care recipient 80. That is, instead of the care recipient 80, the actions (movements) of the caregiver 70 are tracked (detected). As a result, the action of the caregiver 70 moving outside the imaging area while being close to the bed 90 may be erroneously determined as the departure of the care recipient 80. Furthermore, after it is determined that the room is vacated (i.e., empty), the sensitivity of state detection based on the image 1041 may be decreased. This is done to suppress false alarms caused by the erroneous detection of the state of the care recipient 80 even when the room is empty. As a method of decreasing the sensitivity of state detection based on the image 1041, for example, when using the time difference method described above, it is conceivable to decrease the detection sensitivity of the moving object silhouette by adjusting the threshold value in binarization when detecting the moving object silhouette. However, if it is erroneously determined that the care recipient 80 has left the room (the room is empty) despite being in a sleeping state, since the sensitivity of state detection based on the image 1041 has been decreased, there is a possibility that a subsequent fall of the care recipient 80 cannot be detected.

[0068] In the embodiment of FIG. 10, different from the comparative example of FIG. 9, the load on the bed 90 is further detected by the bed sensor 103. Then, since the load is detected by the bed sensor 103, the control unit 101 performs the first control or the second control so as not to notify the detected departure even if the departure of the care recipient 80 is erroneously detected based on the image 1041. As a result, the departure of the care recipient 80 is not transmitted to the server 20. Therefore, false alarms of getting out of bed are suppressed. Further, in the second control, even if a function of reducing the sensitivity of state detection based on the image 1041 is included as a function of the control unit 101 when the departure is determined, false determination of departure is suppressed, so that a missing report of the fall of the care recipient 80 can be prevented. When the third control is performed, as a result, the departure of the care recipient 80 is transmitted to the mobile terminal 30 via the server 20 together with the prohibition information, and thus the departure of the care recipient 80 is not notified in the mobile terminal 30. Similarly, as a result of the departure of the care recipient 80 being transmitted to the mobile terminal 30 via the server 20 together with the undetermined suggestion information, the departure of the care recipient 80 is not notified in the mobile terminal 30.

[0069] When the control unit 101 detects any one of the states of falling, tumbling, and getting out of bed based on the image 1041, the control unit 101 stores an image related to the state. When any one of the states of falling, tumbling, and getting out of bed is detected based on the image 1041, (1) the load on the bed 90 may be erroneously detected by the bed sensor 103, and (2) there is a possibility that the load on the bed 90 is detected because an object is placed on the bed 90. In this case, by storing an image related to a state such as a fall, an actual fall or the like can be recorded, and evidence records of the caregiver 70 entering the room or working can be left.

[0070] When performing the second control, if the control unit 101 detects any of the states of falling, tumbling, or getting out of bed based on the image 1041, it stores the determination result. The determination result may include the following first to third determination results. The control unit 101 can store the first to third determination results separately. The first determination result is, for example, a determination result that the load on the bed 90 is detected by the bed sensor 103 and no falling or the like is detected based on the image 1041. The first determination result corresponds to a determination result that no falling or the like has occurred (for example, sleeping). The second determination result is, for example, a determination result that the load on the bed 90 is not detected by the bed sensor 103 and a falling or the like is detected based on the image 1041. The second determination result corresponds to a determination result that a falling or the like has occurred. The third determination result is a determination result that the load on the bed 90 is detected by the bed sensor 103 and a falling or the like is detected based on the image 1041. The third determination result corresponds to a determination result that a falling or the like is undetermined (there is a possibility of falling or the like) and is distinguished from the second determination result. Thereby, for example, when the notification of falling or the like is suppressed by detecting the load, it is possible to confirm (verify) whether a falling or the like has been detected by the image.

[0071] (Server 20) FIG. 11 is a block diagram showing the hardware configuration of the server 20. The server 20 includes a control unit 201, a communication unit 202, a storage unit 203, and a monitor 204, which are interconnected by a bus. The monitor 204 constitutes a notification unit.

[0072] The control unit 201 includes a CPU, a RAM, a ROM, and the like. The control unit 201 controls each part of the server 20 and performs arithmetic processing according to a program. Details of the functions of the control unit 201 will be described later.

[0073] The communication unit 202 is constituted by an interface circuit for communicating with, for example, a mobile terminal 30 or the like via a LAN.

[0074] The storage unit 203 is constituted by an HDD or an SSD and stores various programs and various data.

[0075] The monitor 204 displays various information such as the status of the care recipient 80 in each room.

[0076] The functions of the control unit 201 will be described.

[0077] The control unit 201 receives, via the communication unit 202, the status of the care recipient 80 from the detection unit 10 as information regarding the status of the person under monitoring. The status of the care recipient 80 includes a specific status. The control unit 201 may further receive, from the detection unit 10, the respiration rate, heart rate, etc. calculated based on the output of the bed sensor 103.

[0078] The control unit 201 displays the status of the care recipient 80 received from the detection unit 10 on the monitor 204.

[0079] FIG. 12 is a diagram showing the management table 2031. The control unit 201 causes the storage unit 203 to pre-store a management table 2031 in which a unique sensor ID for identifying each detection unit 10, the location where each detection unit 10 is installed (room number), and the care recipient 80 who is the person under monitoring in that location are associated with each other.

[0080] In the example of FIG. 12, for example, a sensor ID named SU-1, the location 101 where it is installed, and person A who is the care recipient 80 are associated with each other.

[0081] The control unit 201 refers to the management table 2031 and identifies the installation location of the detection unit 10 and the care recipient 80 who is the person under monitoring based on the sensor ID of the detection unit 10 that transmitted the status of the care recipient 80. The control unit 201 can display the status of the care recipient 80 on the monitor screen of the monitor 204 together with the installation location of the detection unit 10 for each care recipient 80.

[0082] The control unit 201 detects the occurrence of an event based on the state of the care recipient 80 received from the detection unit 10. An event is a change in the state of the care recipient 80 that requires care for the care recipient 80 and is an event for which a report (notification) should be sent to the caregiver 70. The event may include falls, slips, getting out of bed, and leaving the premises, which are specific states. The control unit 201 may detect abnormal breathing and abnormal heartbeats as the occurrence of an event based on the respiration rate and heart rate received from the detection unit 10.

[0083] The control unit 201 manages the addresses of the mobile terminals 30 and may send an event notification for notifying the detected event to the mobile terminals 30 with the managed addresses. The event notification includes information regarding the state of the person being monitored. When the control unit 201 does not manage the addresses of the mobile terminals 30, it may send the event notification without specifying a destination. Note that a port number for each mobile terminal 30 to access the server 20 to obtain the event notification may be preset in the server 20. In this case, in each mobile terminal 30, by inputting the IP address of the server 20 and the set port number, each mobile terminal 30 can access the server 20 and obtain event information from the server 20.

[0084] The control unit 201 may execute, instead of the control unit 101 of the detection unit 10, some or all of the functions of the control unit 101. In this case, the control unit 201 may receive the image 1041 and the output of the bed sensor 103 from the detection unit 10. The control unit 201 may determine the state of the care recipient 80 based on the image 1041 and the output of the bed sensor 103.

[0085] (Mobile Terminal 30) FIG. 13 is a block diagram showing the hardware configuration of the mobile terminal 30. The mobile terminal 30 includes a control unit 301, a wireless communication unit 302, a storage unit 303, an input / output display unit 304, and an audio input / output unit 305, which are interconnected by a bus. The mobile terminal 30 may be configured by a portable communication terminal device such as a tablet computer, a smartphone, or a mobile phone.

[0086] The control unit 301 includes a CPU, a RAM, a ROM, etc. The control unit 301 controls each part of the mobile terminal 30 and performs arithmetic processing according to a program. Details of the functions of the control unit 301 will be described later.

[0087] The wireless communication unit 302 performs wireless communication with each device via the wireless AP 31 or directly by using wireless communication based on standards such as Wi-Fi and Bluetooth (registered trademark). As described above, in the mobile terminal 30, in addition to setting up the wireless LAN, access from the mobile terminal 30 to the server 20 may be enabled by inputting connection information with the server 20 (the IP address of the server 20 and the port number set in the server 20). The connection information is information for identifying the server 20 and may be stored in the storage unit 303.

[0088] The storage unit 303 is composed of an SSD or an SD card and stores various programs and various data.

[0089] The input display unit 304 is composed of, for example, a touch panel in which a touch sensor is superimposed on a display surface such as a liquid crystal. The input display unit 304 displays a terminal monitor screen corresponding to the monitor screen displayed on the monitor 204 of the server 20.

[0090] The voice input / output unit 305 is, for example, a speaker and a microphone, and enables a voice call by the caregiver 70 between the mobile terminal 30 and other mobile terminals 30 via the wireless communication unit 302.

[0091] The functions of the control unit 301 will be described.

[0092] The control unit 301 continuously acquires event notifications from the server 20. The control unit 301 continuously acquires the state of each care recipient 80 from the server 20 as information regarding the state of the person being monitored and displays it on the terminal monitor screen. As described above, the event notification includes information regarding the state of the person being monitored.

[0093] When the state of each care recipient 80 is a specific state, the control unit 301 can notify the state of each care recipient 80 as information regarding the state of the person being monitored by display, voice, vibration, light emission, or a combination thereof on the terminal monitor screen. The functions of voice, vibration, and light emission can be functions that are standard equipment on the mobile terminal 30.

[0094] When the information regarding the state of the person being monitored includes prohibited information or undetermined suggestion information, the control unit 301 does not notify the state of the care recipient 80 as information regarding the state of the person being monitored.

[0095] The operation of the monitoring system 1 will be described.

[0096] FIG. 14 is a flowchart showing the operation of the monitoring system 1. This flowchart can be executed by the control unit 101 of the detection unit 10 according to a program. When the server 20 substitutes for some or all of the functions of the control unit 101 of the detection unit 10, this flowchart can be executed by the control unit 201 of the server 20 according to a program.

[0097] The control unit 101 acquires it by receiving the image 1041 from the camera 104 or the like (S101). The control unit 101 acquires it by receiving the output of the bed sensor 103 or the like (S102).

[0098] The control unit 101 detects the state of the care recipient 80 based on the image 1041 (S103).

[0099] The control unit 101 determines whether the state of the care recipient 80 is a specific state (S104). When the control unit 101 determines that the state of the care recipient 80 is not a specific state (S104: NO), it determines the state of the care recipient 80 detected in step S103 as the state of the care recipient 80 (S106).

[0100] When the control unit 101 determines that the state of the care recipient 80 is a specific state (S104: YES), the control unit 101 determines whether a load is detected based on the output of the bed sensor 103 (S105). When the control unit 101 determines that a load is detected (S105: YES), it executes control not to notify (S107).

[0101] When the control unit 101 determines that no load is detected (S105: NO), it determines the specific state of the care recipient 80 detected in step S103 as the state of the care recipient 80 (S106).

[0102] This embodiment has the following effects.

[0103] Based on at least the image of the area including at least the peripheral area of the bed and the output of the sensor that detects the load on the bed, the state of the person under surveillance is determined, information regarding the state of the person under surveillance based on the determination is notified, and when a predetermined load is detected by the sensor, even if any of falling, toppling, or getting out of bed is detected based on the image, control is performed not to notify the information regarding the state of the person under surveillance. Thereby, false notifications such as the person under surveillance falling or toppling can be suppressed.

[0104] Furthermore, the control not to notify is set as control not to output the information. Thereby, false notifications such as the person under surveillance falling or toppling can be suppressed with a simple configuration.

[0105] Also, the control not to notify is set as control not to notify even when the terminal that notifies the information regarding the state of the person under surveillance has received the information regarding the state of the person under surveillance. Thereby, false notifications such as the person under surveillance falling or toppling can be easily suppressed.

[0106] Also, the control not to notify is set as control not to determine the state detected based on the image as the state of the person under surveillance. Thereby, with a simple configuration, false notifications such as the person under surveillance falling or toppling can be flexibly suppressed.

[0107] Furthermore, when any of the states of falling, tumbling, or getting out of bed is detected based on an image, an image related to the state of the person under monitoring is stored. By doing so, in the case of false detection of load or the like, by storing an image related to the state such as falling, the actual occurrence of a fall or the like can be recorded, and evidence records of the caregiver entering the room or being at work can be left.

[0108] Furthermore, when any of the states of falling, tumbling, or getting out of bed is detected based on an image, the detection result is stored. Thereby, for example, when a load is detected, it is possible to confirm whether a fall or the like has been detected by the image.

[0109] Furthermore, include leaving the room in a specific state. Thereby, false alarms of departure can be suppressed.

[0110] Also, the control not to notify is set as the control to transmit to a terminal that notifies information related to the state of the person under monitoring, including information that prohibits notification of the information related to the state of the person under monitoring. Thereby, false alarms of falls, tumbles, etc. of the person under monitoring can be suppressed more easily.

[0111] Furthermore, the difference between pixels for each adjacent frame of the image is calculated, and the state of the person under monitoring is determined based on the moving object silhouette detected by binarizing the difference. Thereby, the state of the person under monitoring can be detected easily and with high accuracy.

[0112] Furthermore, using a learned model by machine learning, joint points are estimated from the image, and based on the estimated joint points, the state of the person under monitoring is determined. Thereby, the state of the person under monitoring can be detected easily and with higher accuracy.

[0113] Furthermore, when it is determined that the person has left the room, the detection sensitivity of the moving object silhouette is reduced by adjusting the threshold value in the above binarization. Thereby, false alarms when the room is empty can be suppressed.

[0114] Furthermore, the imaging area is set as an area including the area on the bed and the peripheral area of the bed. Thereby, false alarms of falls, tumbles, etc. of the person under monitoring can be further suppressed.

[0115] The configuration of the monitoring system 1 described above explains the main configuration when explaining the features of the above-described embodiments, and is not limited to the above-described configuration, and various modifications can be made within the scope of the claims. For example, in the above-described flowchart, steps other than those shown in the flowchart may be included, or some steps may not be included. Further, the order of the steps is not limited to the above-described embodiments. Furthermore, each step may be executed in combination with other steps as one step, may be executed included in other steps, or may be divided into a plurality of steps and executed.

[0116] In addition, the means and methods for performing various processes in the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a storage device such as a hard disk. Further, the above program may be provided as a single application software, or may be incorporated into the software of the device as one function.

Explanation of Reference Numerals

[0117] 1 Monitoring system 10 Detection unit 101 Control unit 102 Storage unit 103 Bed sensor 104 Camera 1041 Image 20 Server 201 Control unit 202 Communication unit 203 Storage unit 204 Monitor 30 Mobile terminal, 301 Control unit, 302 Wireless communication unit, 303 Memory unit, 304 Input / output display unit, 305 Audio input / output unit, 70 Caregiver, 80 Care recipient, 90 Bed, 91 Peripheral area of the bed.

Claims

1. An imaging unit that images an imaging area including at least the peripheral area of the bed; A sensor that detects the load on the bed; A determination unit that determines the state of the person under surveillance based on the captured image and the output of the sensor; A terminal that notifies information regarding the state of the person under surveillance based on the determination by the determination unit, and has, When it is determined based on the output of the sensor that a predetermined load is detected, control is performed not to notify the information even if any of the states of falling, toppling, or getting out of bed is detected based on the image. A monitoring system.

2. The monitoring system according to claim 1, wherein the control not to notify is that the determination unit does not output the information.

3. The monitoring system according to claim 1, wherein the control not to notify is that the terminal does not notify even if it has received the information.

4. The monitoring system according to claim 1, wherein the control not to notify is that the determination unit does not determine the state detected based on the image as the state of the person under surveillance.

5. It has an image storage unit that stores the image captured by the imaging unit, The monitoring system according to claim 2 or 3, wherein when any of the states of falling, toppling, or getting out of bed is detected based on the image, an image related to the state is stored in the image storage unit.

6. It has a determination result storage unit that stores the determination result of the determination unit, The monitoring system according to any one of claims 2, 3, and 5, wherein when any of the states of falling, toppling, or getting out of bed is detected based on the image, it is stored in the determination result storage unit.

7. The monitoring system according to claim 4, wherein the state further includes leaving the room.

8. The monitoring system according to claim 3, wherein the determination unit includes information prohibiting notification in the information and transmits it to the terminal.

9. The determination unit calculates the difference for each pixel between adjacent frames of the image, and determines the state of the person under surveillance based on the moving object silhouette detected by binarizing the difference. The monitoring system according to any one of claims 1 to 8.

10. The determination unit estimates joint points from the image using a learned model by machine learning, and determines the state of the person under surveillance based on the estimated joint points. The monitoring system according to any one of claims 1 to 8.

11. The monitoring system according to claim 9, which cites claim 7, wherein when the determination unit determines the leaving of the room, the detection sensitivity of the moving object silhouette is reduced by adjusting the threshold value in the binarization.

12. The monitoring system according to any one of claims 1 to 11, wherein the imaging area is an area including the area on the bed and the peripheral area of the bed.

13. An acquisition unit that acquires an image of an imaging area including at least the peripheral area of the bed and an output of a sensor that detects the load on the bed; A determination unit that determines the state of the person being monitored based on the image and the output of the sensor; An informing unit that informs information regarding the state of the person being monitored based on the determination by the determination unit, and has, When it is determined based on the output of the sensor that a predetermined load is detected, even if any of the states of falling, tumbling, or getting out of bed is detected based on the image, a monitoring device that performs control not to inform the information.

14. The monitoring device according to claim 13, wherein the control not to inform is that the determination unit does not output the information.

15. The monitoring device according to claim 13, wherein the control not to inform is that the determination unit does not determine the state detected based on the image as the state of the person being monitored.

16. Having an image storage unit that stores the captured image, The monitoring device according to claim 14, wherein when any of the states of falling, tumbling, or getting out of bed is detected based on the image, an image regarding the state is stored in the image storage unit.

17. Having a determination result storage unit that stores the determination result of the determination unit, The monitoring device according to claim 14 or 16, wherein when any of the states of falling, tumbling, or getting out of bed is detected based on the image, it is stored in the determination result storage unit.

18. The monitoring device according to claim 15, wherein the state further includes leaving the room.

19. The monitoring device according to claim 13, wherein the control not to inform is that the determination unit includes information prohibiting notification in the information and transmits the information regarding the state of the person being monitored based on the determination by the determination unit to a terminal that informs the information.

20. Step (a) of acquiring an image of an imaging area including at least the peripheral area of the bed and an output of a sensor that detects the load on the bed; Step (b) of determining the state of the person being monitored based on the image and the output of the sensor; Step (c) of notifying information regarding the state of the person under surveillance based on the determination in the above step (b); When it is determined based on the output of the sensor that the predetermined load is detected, even if any of the states of falling, tumbling, or getting out of bed is detected based on the image, step (d) of performing control not to notify the information in the above step (c). A monitoring method having this.

21. The monitoring method according to claim 20, wherein the control not to notify is not outputting the information in the above step (b).

22. The monitoring method according to claim 20, wherein the control not to notify is not determining the state detected based on the image as the state of the person under surveillance in the above step (b).

23. The monitoring method according to claim 21, having step (e) of storing an image regarding the state when any of the states of falling, tumbling, or getting out of bed is detected based on the image.

24. The monitoring method according to claim 21 or 23, having step (f) of storing the determination result in the above step (b) when any of the states of falling, tumbling, or getting out of bed is detected based on the image.

25. The monitoring method according to claim 22, wherein the state further includes leaving the room.

26. The monitoring method according to claim 20, wherein the control not to notify is including information prohibiting notification in the above step (b) and transmitting it to a terminal that notifies information regarding the state of the person under surveillance based on the determination.

27. A monitoring program for causing a computer to execute the monitoring method according to any one of claims 20 to 26.

Images