Monitoring system, monitoring method, and monitoring program

Abstract

To provide a monitor system, monitor method, and monitor program which can prevent or suppress a false alarm or missing alarm of a movement of a care-receiver who is required give alarm such as his / her slip-down, overturn, and the like.SOLUTION: A monitor system 1 includes an imaging part 103, a bed sensor 95, and a determination part. The imaging part images an image region including at least a bed. The bed sensor 95 detects a load on the bed, The determination part determines an object that has been fallen-out from the bed on the basis of, the information obtained by the imaging part and the information obtained by the bed sensor 95.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring method, and a monitoring program. [Background technology]

[0002] In nursing care facilities such as nursing homes for the elderly and hospitals, residents or hospitalized care recipients (hereinafter referred to as "care recipients" or "users") may fall while walking or fall out of bed and injure themselves within the facility. Therefore, development is underway to develop monitoring systems that can constantly monitor the condition of care recipients so that nurses, caregivers, etc. (hereinafter referred to as "caregivers") can rush to their aid when such a situation occurs.

[0003] The following Patent Document 1 discloses a technology that detects whether a monitored person has "slid off" from a bed based on images. With this technology, if a monitored person is in a lying position with either the upper or lower half of their body protruding from the bedside, and it is detected that 30% or more but less than 100% of the volume of a moving object exceeds the bed frame, it is determined that the person has "slid off." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 030880 Summary of the Invention [Problem to be solved by the invention]

[0005] After careful consideration by the inventors, the technology of Patent Document 1 was found to have the following problems: if an object other than the person being monitored (person being cared for), such as a futon, sticks out from the bed, it may be mistakenly detected as the person being monitored having slipped off; and if the person being monitored slides off the bed very slowly, it may not be recognized as a moving object, resulting in a missed alarm.

[0006] The present invention has been made to solve these problems, and aims to provide a monitoring system, a monitoring method, and a monitoring program that can prevent or suppress false or missed reports about conditions of a care recipient that require notification, such as slipping, falling, or tripping. [Means for solving the problem]

[0007] The above-mentioned problems of the present invention are solved by the following means.

[0008] (1) A surveillance system comprising an imaging unit that captures an image of a shooting area including at least a bed, a bed sensor that detects the load on the bed, and a determination unit that determines whether an object has protruded from the bed based on information obtained from the imaging unit and information obtained from the bed sensor.

[0009] (2) The monitoring system described in (1) above, wherein the determination unit determines whether the object protruding from the bed is only a futon or includes at least part of a body based on information obtained from the imaging unit and information obtained from the bed sensor.

[0010] (3) The monitoring system according to (2) above, further comprising a control unit that performs control not to notify the determination result when the object protruding from the bed is determined to be only a futon.

[0011] (4) The monitoring system described in (2) above, further comprising a control unit that controls to notify the determination result when it is determined that the object protruding from the bed includes at least part of a body.

[0012] (5) The monitoring system according to any one of (1) to (4) above, wherein the information obtained from the imaging unit is information obtained by analyzing an image captured by the imaging unit.

[0013] (6) The monitoring system according to any one of (1) to (5) above, wherein the information obtained from the bed sensor is information obtained by analyzing the load detected by the bed sensor.

[0014] (7) The monitoring system according to any one of (1) to (4) and (6) above, wherein the information obtained from the imaging unit is image information captured by the imaging unit.

[0015] (8) The monitoring system according to any one of (1) to (5) and (7) above, wherein the information obtained from the bed sensor is load information detected by the bed sensor.

[0016] (9) The monitoring system described in (7) above further includes an acquisition unit that acquires information from the bed sensor, changes the image recognition sensitivity based on the information obtained from the bed sensor, and then acquires the image information.

[0017] (10) The monitoring system according to (7) above, further comprising an acquisition unit that acquires the image information from the imaging unit installed on the bed.

[0018] (11) The monitoring system according to any one of (1) to (10) above, further comprising an output unit that outputs the determination result of the determination unit.

[0019] (12) A monitoring method including an imaging step in which an imaging unit images an imaging area including at least a bed, a detection step in which a bed sensor detects the load on the bed, and a determination step in which an object protruding from the bed is determined based on information obtained from the imaging unit and information obtained from the bed sensor.

[0020] (13) The monitoring method described in (12) above, wherein in the determination step, it is determined whether the object protruding from the bed is only a futon or includes at least part of a body based on the information obtained from the imaging unit and the information obtained from the bed sensor.

[0021] (14) The monitoring method described in (13) above further includes a control step of performing control not to notify the judgment result if the judgment step determines that the object protruding from the bed is only a futon.

[0022] (15) The monitoring method described in (13) above further includes a control step of performing control to notify the judgment result when it is determined in the judgment step that the object protruding from the bed includes at least part of a body.

[0023] (16) The monitoring method according to any one of (12) to (15) above, wherein the information obtained from the imaging unit is information obtained by analyzing an image captured by the imaging unit.

[0024] (17) The monitoring method according to any one of (12) to (16) above, wherein the information obtained from the bed sensor is information obtained by analyzing the load detected by the bed sensor.

[0025] (18) The monitoring method according to any one of (12) to (15) and (17) above, wherein the information obtained from the imaging unit is image information captured by the imaging unit.

[0026] (19) The monitoring method according to any one of (12) to (16) and (18) above, wherein the information obtained from the bed sensor is load information detected by the bed sensor.

[0027] (20) The monitoring method according to (18) above, further comprising acquiring information from the bed sensor, changing the image recognition sensitivity based on the information obtained from the bed sensor, and then acquiring the image information.

[0028] (21) The monitoring method according to any one of (12) to (20) above, further comprising an output step of outputting the determination result in the determination step.

[0029] (22) A monitoring program for causing a computer to execute the monitoring method according to any one of (12) to (21) above. [Effects of the Invention]

[0030] According to the present invention, an object protruding from the bed is determined based on information obtained from the imaging unit and information obtained from the bed sensor, thereby preventing or suppressing false or missed alerts regarding the condition of the person being cared for that requires notification, such as when the person being cared for slips or falls off the bed. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a diagram showing the overall configuration of a monitoring system according to a first embodiment. [Figure 2] 2 is a schematic diagram illustrating a bed for a care recipient on which the detection device shown in FIG. 1 is installed. [Figure 3] FIG. 3 is a schematic diagram of the bed of the care recipient shown in FIG. 2 as viewed from the side. [Figure 4] FIG. 2 is a block diagram showing a hardware configuration of the detection device shown in FIG. [Figure 5] 5 is a block diagram illustrating the main functions of the arithmetic and control unit shown in FIG. 4. [Figure 6] FIG. 2 is a block diagram showing the hardware configuration of the server shown in FIG. [Figure 7] 4 is a flowchart illustrating a processing procedure of a monitoring method for a detection device according to the first embodiment. [Figure 8] 10A and 10B are diagrams illustrating an example of determining whether a person is protruding from a bed based on image information and bed sensor information. [Figure 9] FIG. 10 is a schematic diagram illustrating an example of an image when a care recipient is lying down. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of an image when a care recipient gets out of bed. [Figure 11] 10 is a schematic diagram illustrating an example of an image in which a futon object protrudes from a predetermined area. FIG. [Figure 12]FIG. 10 is a schematic diagram illustrating an example of an image in which a part (feet) of a care-receiver object protrudes from a predetermined area. [Figure 13] FIG. 10 is a schematic diagram illustrating an example of an image when a care recipient gets out of bed. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of an image when a care recipient slips down. [Figure 15] FIG. 10 is a schematic diagram illustrating an example of an image in which the care recipient is lying down as an initial state. [Figure 16] 10A and 10B are diagrams illustrating an example of determining whether a person is protruding from a bed based on image information and bed sensor information. [Figure 17] 10 is a schematic diagram illustrating an example of an image when a care recipient falls or trips. FIG. [Figure 18] 10 is a schematic diagram illustrating an example of an image in which a futon object protrudes from a predetermined area. FIG. [Figure 19] 10 is a flowchart illustrating a processing procedure of a monitoring method by a detection device according to a third embodiment. [Figure 20] FIG. 10 is a schematic diagram illustrating an example of an image in which the care recipient is lying down as an initial state. [Figure 21] 10 is a diagram illustrating an example of determining the state of a care recipient based on image information and bed sensor information. FIG. [Figure 22] FIG. 10 is a schematic diagram illustrating an example of an image when a care recipient gets out of bed. [Figure 23] 10 is a schematic diagram illustrating an example of an image when a care recipient falls or trips. FIG. [Figure 24] FIG. 10 is a schematic diagram illustrating an example of an image in which a care recipient has left the bed, as a comparative example. [Figure 25] As a comparative example, this is a schematic diagram illustrating an image in which a care recipient has fallen or tumbled. DETAILED DESCRIPTION OF THE INVENTION

[0032] A monitoring system, a monitoring method, and a monitoring program according to an embodiment of the present invention will be described below with reference to the drawings. In the drawings, identical elements are designated by the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0033] (First embodiment) FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 according to a first embodiment. In this embodiment, the monitoring system 1 is used in a nursing care facility or the like. The monitoring system 1 includes a detection device 10, a server 20, a plurality of mobile terminals 30, and a wireless AP (access point) 31. The detection device 10, the server 20, and the mobile terminals 30 are connected to each other so as to be able to communicate with each other via a LAN (local area network) and the wireless AP 31. The mobile terminal 30 is carried by a caregiver 70 who is caring for or providing care to a care recipient 80.

[0034] (90 beds) Fig. 2 is a schematic diagram illustrating a bed 90 for a care recipient 80 on which the detection device 10 shown in Fig. 1 is installed. Fig. 3 is a schematic diagram of the bed 90 shown in Fig. 2 viewed from the side. In Fig. 3, the direction from left to right on the paper (the direction from the head side to the foot side of the bed 90) is defined as the X-axis direction, the direction from the front to the back of the paper (the width direction of the bed 90) is defined as the Y-axis direction, and the direction from bottom to top of the paper (the height direction of the bed 90) is defined as the Z-axis direction.

[0035] 2 and 3, the bed 90 includes a frame 91, a headboard 92, a footboard 93, a mattress 94, and a bed sensor 95. A futon 96 is placed on the mattress 94. The futon 96 may include bedding such as a comforter, a blanket, and a towel blanket. The bed 90 may be, for example, an electric bed.

[0036] The bed sensor 95 is a load sensor that detects the pressure caused by the weight of the care recipient 80 on the bed 90. In this way, the bed sensor 95 detects whether or not the care recipient 80 is on the bed 90. The bed sensor 95 may be composed of a sheet-like bag filled with air and a piezoelectric sensor that detects fluctuations in the air pressure. The bed sensor 95 may be placed, for example, on a portion of the mattress 94 that comes into contact with the back, waist, buttocks, etc. of the care recipient 80. The detection results of the bed sensor 95 (bed sensor information) are transmitted to the calculation control unit 101. The bed sensor information may be information (load sensor information) related to the load measured by the load sensor. In this embodiment, the bed sensor information indicates, for example, whether or not the care recipient 80 is carrying a load, with "weighted" and "unweighted," respectively.

[0037] (Detection device 10) Fig. 4 is a block diagram showing the hardware configuration of the detection device 10 shown in Fig. 1. The detection device 10 has an arithmetic and control unit 101, a communication unit 102, and a camera 103, which are interconnected by a bus. The detection device 10 is installed, for example, on the upper end of a headboard 92 of a bed 90.

[0038] The arithmetic and control unit 101 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. The arithmetic and control unit 101 may also include an HDD or SSD (Solid State Drive). The arithmetic and control unit 101 controls each unit of the detection device 10 and performs arithmetic processing in accordance with a monitoring program. The main functions of the arithmetic and control unit 101 will be described later.

[0039] The communication unit 102 is an interface circuit (for example, a LAN card, a wireless communication circuit, etc.) for communicating with, for example, the server 20 via a LAN.

[0040] Camera 103 captures an image within a monitoring area including at least bed 90 and the area surrounding bed 90, and outputs the captured image (image information). Camera 103 constitutes an imaging unit. This image information includes still images and video. Video may include multiple still images captured intermittently in time series. Camera 103 is a near-infrared camera, but a visible light camera may be used instead, or both may be used.

[0041] Fig. 5 is a block diagram illustrating the main functions of the arithmetic and control unit 101 shown in Fig. 1. The arithmetic and control unit 101 functions as an acquisition unit 111, a determination unit 112, and an output unit 113.

[0042] The acquisition unit 111 acquires image information and bed sensor information. The determination unit 112 determines whether the object protrudes from the bed 90 based on the image information and the bed sensor information. A method for determining whether the object protrudes from the bed 90 will be described in detail later. The output unit 113 outputs the determination result of the determination unit 112. The output from the output unit 113 is transmitted to the mobile terminal 30 by the communication unit 102.

[0043] (Server 20) 6 is a block diagram showing the hardware configuration of the server 20. The server 20 has an arithmetic and control unit 201, a communication unit 202, and a storage unit 203, which are interconnected by a bus.

[0044] The arithmetic and control unit 201 includes a CPU, RAM, ROM, etc. The arithmetic and control unit 201 controls each unit of the server 20 and performs arithmetic processing in accordance with a program. The arithmetic and control unit 201 can also be configured to cooperate with the determination unit 112 to determine whether the bed 90 is protruding.

[0045] The communication unit 202 is configured with an interface circuit for communicating with, for example, the detection device 10 and the mobile terminal 30 via a LAN. The communication unit 202 receives the protrusion determination result from the detection device 10. Furthermore, when the arithmetic and control unit 201 performs protrusion determination, the communication unit 202 receives image information and bed sensor information from the detection device 10. The protrusion determination result is stored in the storage unit 203.

[0046] The storage unit 203 is configured by an HDD or SSD, and stores various programs and various data.

[0047] (Monitoring method) FIG. 7 is a flowchart illustrating the processing steps of the monitoring method using the detection device in the first embodiment. The processing of the flowchart in this figure is realized by the CPU of the arithmetic and control unit 101 executing a monitoring program. FIG. 8 is a diagram illustrating an example of protrusion determination from the bed 90 based on image information and bed sensor information. FIG. 9 is a schematic diagram illustrating an example of an image in which the care recipient 80 is lying down. Note that, for the sake of simplicity, an object on the image corresponding to the footboard 93 is omitted in FIG. 9 (the same applies to the following FIGS. 10 to 14).

[0048] As shown in FIG. 7, first, the camera 103 starts capturing images within the monitoring area (step S101). The calculation and control unit 101 controls the camera 103 to start (or continue) capturing images within the monitoring area. In this embodiment, as an example, it is assumed that the care recipient 80 who is lying in bed 90 gets up from the bed 90, gets off the floor and tries to walk. That is, the initial state of the care recipient 80 is lying in bed, and then transitions to getting out of bed. The camera 103 captures the care recipient 80 and the bed 90 within the monitoring area.

[0049] Next, image information and bed sensor information are acquired (step S102). The acquisition unit 111 acquires the image captured by the camera 103 and transmits it as image information to the determination unit 112. The acquisition unit 111 also transmits the bed sensor information to the determination unit 112.

[0050] Next, protrusion from the bed 90 is determined (step S103). The determination unit 112 detects a moving object from the image information acquired by the acquisition unit 111. A moving object can be detected, for example, by extracting a range of pixels with a relatively large difference using a time subtraction method that extracts the difference between images (frames) captured at different times. A moving object can also be detected using a background subtraction method that extracts the difference between a captured image and a background image. In this embodiment, it is assumed that the care recipient 80 is detected as a moving object as the care recipient 80 changes state from lying down to getting out of bed. Furthermore, if the care recipient 80 is covered with a futon 96, the futon 96 may move. Therefore, in this embodiment, it is considered that the care recipient 80 and / or the futon 96 may be detected as a moving object. Note that in this embodiment, the bed 90 may be an electric bed, but while the care recipient 80 is being monitored, operations such as reclining are stopped and the mattress 94 is not moved.

[0051] However, it is difficult to distinguish between a state in which the futon 96 is protruding from the bed 90 and a state in which the care recipient 80 has slipped off, based solely on the detection result of a moving object obtained from image information. On the other hand, it is difficult to distinguish between a state in which a part of the body of the care recipient 80 is protruding from the bed 90 and a state in which the care recipient 80 is lying down in the bed 90, based solely on bed sensor information. Therefore, in this embodiment, the determination unit 112 determines whether an object is protruding from the bed 90, based on both the image information and the bed sensor information. Here, protruding from the bed 90 means that a part of the body of the care recipient 80 and / or the futon 96 is protruding from the bed 90. The determination unit 112 then determines whether an object is protruding from the bed 90, based on the image information (information obtained from the camera 103) and the bed sensor information (information obtained from the bed sensor 95). In the following, the object of the care recipient 80 on the image will be referred to as a care recipient object 800, the object of the mattress 94 as a mattress object 940 (the shaded area in Figure 9), the object of the bed sensor 95 as a bed sensor object 950, and the object of the futon 96 as a futon object 960.

[0052] (1) Judgment pattern 1 As shown in FIG. 9 , when there is no change in the image around a predetermined area A (a photographed area indicated by a dashed line) including the mattress object 940 (i.e., no object moving from the inside to the outside of the predetermined area) and when the bed sensor information from the bed sensor 95 indicates "weighted" (the bed sensor object 950 is shown in light gray), the determination unit 112 determines that there is "no protrusion" from the bed 90. More specifically, the determination unit 112 detects (determines) the presence or absence of a load by detecting whether the load exceeds a predetermined threshold value related to the load set corresponding to the weight of each care recipient 80. The predetermined threshold value related to the load is set in advance for each care recipient 80 and can be stored in the RAM of the arithmetic and control unit 101. The predetermined area A is set to be slightly larger than the mattress object 940. That is, the outer edge of the predetermined area A can be set along the outer edge of the mattress object 940 at a position m away from the outer edge of the mattress object 940. This is because it is expected that the futon object 960 will protrude from the mattress object 940 to some extent even when the futon 96 is properly placed over the care recipient 80 lying in bed.

[0053] 9 illustrates only a portion of the predetermined area A for the sake of simplicity, but the outer edge of the mattress object 940 outside the monitoring area may also be set at a position m away from the outer edge of the mattress object 940. The setting of the outer edge of the predetermined area A is similar in the following FIGS. 10 to 14.

[0054] Furthermore, the determining unit 112 determines that the care recipient 80 is lying down because there is no protrusion from the bed 90 (see FIG. 8).

[0055] (2) Judgment pattern 2 10 is a schematic diagram illustrating an example of an image in which the care recipient 80 has left the bed. When there is no change in the image around the predetermined area A (no moving object is detected by the time subtraction method or the background subtraction method) and the bed sensor information from the bed sensor 95 is "no weight," the determination unit 112 determines that there is "no protrusion" from the bed 90. For example, if the care recipient 80 moves slowly, there may be no change in the image around the predetermined area A. Even if no moving object can be detected from the image information, the determination unit 112 determines that the care recipient 80 has left the bed because the bed sensor information is "no weight."

[0056] (3) Judgment pattern 3 FIG. 11 is a schematic diagram illustrating an image in which the futon object 960 protrudes from the predetermined area A, and FIG. 12 is a schematic diagram illustrating an image in which a part (feet) of the care recipient object 800 protrudes from the predetermined area A. If there is a change in the image around the predetermined area A and the bed sensor information from the bed sensor 95 is "weighted," the determination unit 112 determines that "an object is protruding." If nothing other than the care recipient object 800 and the futon object 960 protrudes from the predetermined area A, it is estimated that either the care recipient object 800 or the futon object 960, or both, have protruded. Therefore, the determination unit 112 determines that the state of the care recipient 80 is lying down or that part of the body is protruding, since the state is at least not slipping or getting out of bed.

[0057] (4) Judgment pattern 4 Fig. 13 is a schematic diagram illustrating an image when the care recipient 80 gets out of bed, and Fig. 14 is a schematic diagram illustrating an image when the care recipient 80 has slipped down. If there is a change in the image around the predetermined area A and the bed sensor information from the bed sensor 95 is "no weight", the determination unit 112 determines that "the care recipient has protruded" from the bed 90. Furthermore, the determination unit 112 determines that the care recipient 80 has gotten out of bed or slipped down because the care recipient object 800 has moved outside the predetermined area A and is not present on the mattress object 940.

[0058] Next, the determination result is output (step S104). The output unit 113 outputs the determination result of protrusion from the bed 90 and / or the determination result of the condition of the care recipient 80. The determination result is transmitted to the mobile terminal 30 via the server 20. For example, the output unit 113 can be configured not to output the determination result when the care recipient 80 is lying down or when part of his / her body protrudes from the bed, but to output the determination result when the care recipient 70 needs to rush to the care recipient 80 as soon as possible due to a slip, tumble, or fall, etc. Furthermore, in the case of a slip, tumble, or fall, the output unit 113 can output the determination result, as well as send an instruction to the mobile terminal 30 to sound an alarm, or send an image to the mobile terminal 30 via the server 20 so that the care recipient 70 can check the condition of the care recipient 80. Furthermore, the output unit 113 may be configured to output the determination result in the case of getting out of bed, and to notify the caregiver 70 according to the level of care required of the care recipient 80. Furthermore, the output unit 113 may be configured to output the determination result and notify the caregiver 70, for example, when a part of the body of the care recipient 80 or the futon 96 has been sticking out for a long period of time, from the perspective of the safety and physical condition of the care recipient 80.

[0059] In this way, in the processing of the monitoring method of the flowchart shown in FIG. 7, image information and bed sensor information are acquired, and protrusion from the bed 90 is determined based on both the image information and the bed sensor information.

[0060] The detection device 10 of this embodiment described above determines whether an object has protruded from the bed 90 based on information obtained from the camera 103 and information obtained from the bed sensor 95, and therefore can prevent or suppress false or missed alerts regarding the condition of the person being cared for 80 that requires notification, such as when the person being cared for 80 slips or falls off the bed 90.

[0061] (Second embodiment) In the second embodiment, a case will be described in which, when an object in an image moves downward outside the predetermined area A, protrusion from the bed 90 is determined based on bed sensor information. Note that, in the following, detailed description of the same configuration as in the first embodiment will be omitted to avoid duplication of explanation.

[0062] Fig. 15 is a schematic diagram illustrating an image of the care recipient 80 lying down in bed as an initial state, and Fig. 16 is a diagram illustrating a determination of protrusion from the bed 90 based on image information and bed sensor information. Fig. 17 is a schematic diagram illustrating an image of the care recipient 80 tumbling or falling, and Fig. 18 is a schematic diagram illustrating an image of the futon object 960 when part of the futon object 960 protrudes from the predetermined area A. To simplify the explanation, the object corresponding to the footboard 93 is omitted from Figs. 15, 17, and 18.

[0063] In this embodiment, a threshold value TH is set to distinguish whether the object has moved upward or downward outside the predetermined area A. The threshold value TH is set, for example, to the position of the upper end of the mattress object 940 on the image. If the position of the object in the vertical direction on the image exceeds the threshold value TH, the determination unit 112 determines that the object has moved upward outside the predetermined area A on the image. On the other hand, if the position of the object is less than the threshold value TH, the determination unit 112 determines that the object has moved downward.

[0064] As shown in Figure 16, if there is a change in the image below and outside the specified area A and the bed sensor information from the bed sensor 95 is "no weight", the judgment unit 112 judges that "the person being cared for is protruding from the bed 90."

[0065] Furthermore, as shown in Figure 17, the judgment unit 112 determines that the cared-for person object 800 on the image has moved outside the specified area A and is not present on the mattress object 940, and therefore determines that the cared-for person 80 has fallen or fallen.

[0066] On the other hand, if there is a change in the image below and outside the predetermined area A and the bed sensor information from the bed sensor 95 is "weighted," the determination unit 112 determines that "an object is protruding" from the bed 90. If the only objects protruding from the predetermined area A are the care recipient object 800 and the futon object 960, it is estimated that either the care recipient object 800 or the futon object 960, or both, have protruded. Figure 18 illustrates an example in which the futon object 960 protrudes from the predetermined area A.

[0067] As described above, in this embodiment, when an object in the image moves downward outside the predetermined area A, it is determined that the object has protruded from the bed 90 based on the bed sensor information. Therefore, the determination unit 112 can determine whether an object that has moved downward outside the predetermined area A is due to the object protruding from the care recipient 80 or the futon 96. Furthermore, the determination unit 112 can determine whether the state of the care recipient 80 is a tumble or fall, or whether the object is lying in bed (not a tumble or fall).

[0068] (Third embodiment) In the third embodiment, when the bed sensor information indicates "no weight," the state of the care recipient 80 is determined based on whether the object in the image has moved upward or downward outside the predetermined area A. In the following, detailed description of the same configuration as in the first embodiment will be omitted to avoid duplication.

[0069] FIG. 19 is a flowchart illustrating the processing steps of the monitoring method using the detection device 10 in the third embodiment. The processing of the flowchart in FIG. 19 is realized by the CPU of the arithmetic and control unit 101 executing a monitoring program. FIG. 20 is a schematic diagram illustrating an image of the care recipient 80 lying down as an initial state, and FIG. 21 is a diagram illustrating an example of the state determination of the care recipient 80 based on image information and bed sensor information. FIG. 22 is a schematic diagram illustrating an image of the care recipient 80 getting out of bed, and FIG. 23 is a schematic diagram illustrating an image of the care recipient 80 having tumbled or fallen. FIG. 24 is a schematic diagram illustrating an image of the care recipient 80 getting out of bed as a comparative example, and FIG. 25 is a schematic diagram illustrating an image of the care recipient 80 having tumbled or fallen as a comparative example. For ease of explanation, the object corresponding to the footboard 93 is omitted from FIGS. 20, 22 to 25.

[0070] As shown in Fig. 19, first, the camera 103 starts capturing images within the monitoring area (step S201). The arithmetic and control unit 101 controls the camera 103 to start (or continue) capturing images within the monitoring area. In this embodiment, as in the second embodiment, the initial state is assumed to be that the care recipient 80 is lying down, and a threshold value TH is set to distinguish whether an object that has moved outside the predetermined area A has moved upward or downward outside the predetermined area A (see Fig. 20).

[0071] Next, the bed sensor information is acquired (step S202). The acquisition unit 111 transmits the detection result of the bed sensor 95 to the determination unit 112 as bed sensor information.

[0072] Next, it is determined whether the bed sensor information is "unweighted" (step S203). As shown in FIG. 21, if the bed sensor information is "unweighted" (step S203: YES), the image recognition sensitivity is changed (step S204). The arithmetic and control unit 101 controls the camera 103 to increase the image recognition sensitivity around the bed 90. For example, the arithmetic and control unit 101 controls the camera 103 to increase the frame rate of image capture, thereby increasing the image recognition sensitivity. Increasing the frame rate improves the image quality and the accuracy of detecting the position of an object in the image. On the other hand, if the bed sensor information is not "unweighted," that is, "weighted" (step S203: NO), the image recognition sensitivity is not changed and the process proceeds to the next step S205. Note that if the bed sensor information is "weighted," the care recipient 80 has not moved from the bed 90, so there is no need to increase the image recognition sensitivity.

[0073] Next, image information is acquired (step S205). The acquisition unit 111 acquires the image captured by the camera 103 and transmits it to the determination unit 112 as image information.

[0074] Next, the state of the care recipient 80 is determined (step S206). As shown in Fig. 22, the determination unit 112 determines that the care recipient 80 has gotten out of bed (not fallen over or down) if there is any object that has moved upward outside the predetermined area A. Also, as shown in Fig. 23, the determination unit 112 determines that the care recipient 80 has fallen over or down if there is any object that has moved downward outside the predetermined area A and there is no object that has moved upward.

[0075] On the other hand, as shown in Fig. 24, if the bed sensor 95 is not placed on the mattress 94, bed sensor information cannot be obtained, and therefore the image recognition sensitivity is not changed. Therefore, the determination unit 112 may not be able to detect even if the care recipient 80 gets out of bed. Similarly, as shown in Fig. 25, if the bed sensor 95 is not placed on the mattress 94, it may not be able to detect even if the care recipient 80 falls or tumbles.

[0076] Next, the determination result is output (step S207). The output unit 113 outputs the determination result of the state of the care recipient 80. The determination result is transmitted to the mobile terminal 30 via the server 20.

[0077] In this way, in the processing of the flowchart shown in Figure 19, bed sensor information is acquired, and if the bed sensor information is "unweighted," the image recognition sensitivity is changed, image information is acquired, and the condition of the care recipient 80 is determined based on the image information and the bed sensor information.

[0078] According to the detection device 10 of this embodiment described above, it is possible to prevent or suppress false or missed reports about the condition of the person being cared for 80 that requires notification, such as when the person being cared for 80 slips off the bed or falls.

[0079] The configuration of the monitoring system 1 described above is a main configuration that is used to explain the features of the above-described embodiment, and is not limited to the above-described configuration, and various modifications are possible within the scope of the claims.

[0080] For example, in the above-described first to third embodiments, the case where the detection device 10 is installed on the headboard 92 has been described, but the detection device 10 may be installed in a location other than the headboard 92, such as the footboard 93. Furthermore, the detection device 10 is not limited to being installed on the upper end of the headboard 92 or footboard 93, and may also be installed, for example, in a plane facing the center of the bed 90.

[0081] Furthermore, in the above-described first to third embodiments, the determination unit 112 determines whether an object has protruded from the bed 90 based on whether there is a change in the periphery of the predetermined region A in the image information and whether there is a weighting of the bed sensor information. However, the present invention is not limited to such a case, and the determination unit 112 may be configured to analyze the image captured by the camera 103 using a known image recognition technique, pattern matching technique, machine learning technique, or the like, and determine whether there is a protrusion of an object from the bed 90 based on the image analysis result and whether there is a weighting of the bed sensor information.

[0082] Possible objects protruding from the bed 90 include only the futon 96, the futon 96 and a body part of the care recipient 80, or only a body part of the care recipient 80. When performing image analysis using machine learning technology, the determination unit 112 can determine the protruding object, for example, as follows. Conditions such as the position and state of the futon 96 on the bed 90 are changed in advance, and images of a predetermined area A and its surroundings are taken when only the futon 96 protrudes from the bed 90. The captured images and correct labels are input as training data into a deep neural network for learning. In this case, the correct labels are set to a 100% probability (likelihood) that only the futon 96 protrudes from the bed 90, a 0% probability that the futon 96 and a body part of the care recipient 80 protrude, and a 0% probability that only a body part of the care recipient 80 protrudes. When monitoring the care recipient 80, the determination unit 112 inputs the image (information) obtained from the camera 103 into a deep neural network and obtains, as an analysis result, the probability that only the futon 96 is sticking out from the bed 90, etc. Based on the analysis result and bed sensor information (information obtained from the bed sensor 95), the determination unit 112 determines whether the object sticking out from the bed 90 is only the futon 96 or whether it includes at least a part of the body. For example, if the probability that only the futon 96 is sticking out from the bed 90 is equal to or greater than a predetermined value and the bed sensor information is "weighted," the determination unit 112 determines that the object sticking out from the bed 90 is only the futon 96. On the other hand, if the probability that only the futon 96 is sticking out from the bed 90 is less than a predetermined value and the bed sensor information is "weighted," the determination unit 112 determines that the object sticking out from the bed 90 includes at least a part of the body of the care recipient 80.

[0083] When the determination unit 112 determines that the object protruding from the bed 90 is only the futon 96, the output unit 113 performs control not to output (notify) the determination result. This makes it possible to prevent a false alarm from being issued about the condition of the care recipient 80 that does not require notification. As a result, the burden on the caregiver 70 can be reduced. On the other hand, when the determination unit 112 determines that the object protruding from the bed 90 includes at least a part of the body, the output unit 113 performs control to output the determination result.

[0084] Furthermore, although the case where the determination unit 112 performs image analysis has been described, the present invention is not limited to such a case. For example, the camera 103 can be configured to analyze the image of the predetermined area A and its surroundings to determine whether an object protrudes from the bed 90, and the determination unit 112 can determine whether an object protrudes from the bed 90 based on the determination result by the camera 103 and bed sensor information. Alternatively, the image (image information) captured by the camera 103 can be sent directly to the server 20 without analysis, and the server 20 can analyze the image to determine whether an object protrudes from the bed 90, and can determine whether an object protrudes from the bed 90 based on the determination result and bed sensor information.

[0085] It is also possible to configure the bed sensor 95 to detect the load (load information) on the bed 90, analyze the detected load information to determine whether an object has protruded from the bed 90, and the determination unit 112 to determine whether an object has protruded from the bed 90 based on the analysis results of the image information and the load information.It is also possible to configure the bed sensor 95 to send the load information detected by the bed sensor 95 directly to the server 20 without analyzing it, and the server 20 to analyze the load information on the bed 90 to determine whether an object has protruded from the bed 90, and to determine whether an object has protruded from the bed 90 based on the analysis results of the image information and the load information.

[0086] Furthermore, the above-described flowcharts may include steps other than those shown in the flowcharts, or some steps may not be included. The order of the steps is not limited to the above-described embodiment. Furthermore, each step may be combined with other steps and executed as a single step, may be included in other steps and executed, or may be divided into multiple steps and executed.

[0087] The means and methods for performing various processes in the above-described embodiments can be realized by either dedicated hardware circuits or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a device such as a detection unit. [Explanation of symbols]

[0088] 1. Surveillance system, 10 detection device, 101 arithmetic and control unit, 102 Communications Department, 103 cameras, 111 Acquisition Department; 112 Judgment Department, 113 output section, 20 servers, 201 arithmetic and control unit, 202 Communications Department, 203 Memory section, 30 mobile devices, 31 access points, 90 beds, 91 Frame part, 92 headboard, 93 footboard, 94 mattresses, 95 sensor unit, 96 futons.

Claims

1. an imaging unit that captures an imaging area including at least the bed; a bed sensor for detecting a load on the bed; A monitoring system comprising: a determination unit that determines whether an object has protruded from the bed based on information obtained from the imaging unit and information obtained from the bed sensor.

2. The monitoring system of claim 1 , wherein the determination unit determines whether the object protruding from the bed is only a futon or includes at least part of a body based on information obtained from the imaging unit and information obtained from the bed sensor.

3. The monitoring system according to claim 2 , further comprising a control unit that, when the object protruding from the bed is determined to be only a futon, performs control not to notify the determination result.

4. The monitoring system according to claim 2 , further comprising a control unit that performs control to notify a user of a determination result when it is determined that the object protruding from the bed includes at least a part of a body.

5. 5. The monitoring system according to claim 1, wherein the information obtained from the imaging unit is information obtained by analyzing an image captured by the imaging unit.

6. 6. The monitoring system according to claim 1, wherein the information obtained from the bed sensor is information obtained by analyzing the load detected by the bed sensor.

7. 7. The monitoring system according to claim 1, wherein the information obtained from the imaging unit is image information captured by the imaging unit.

8. 8. The monitoring system according to claim 1, wherein the information obtained from the bed sensor is load information detected by the bed sensor.

9. The monitoring system according to claim 7 , further comprising an acquisition unit that acquires information from the bed sensor, changes image recognition sensitivity based on the information obtained from the bed sensor, and then acquires the image information.

10. The monitoring system according to claim 7 , further comprising an acquisition unit that acquires the image information from the imaging unit installed on the bed.

11. The monitoring system according to any one of claims 1 to 10, further comprising an output unit that outputs the determination result of the determination unit.

12. an imaging step in which the imaging unit images an imaging area including at least the bed; a detecting step in which a bed sensor detects a load on the bed; A monitoring method including a determination step of determining whether an object has protruded from the bed based on information obtained from the imaging unit and information obtained from the bed sensor.

13. The monitoring method according to claim 12, wherein in the determining step, it is determined whether the object protruding from the bed is only a futon or includes at least part of a body based on the information obtained from the imaging unit and the information obtained from the bed sensor.

14. 14. The monitoring method according to claim 13, further comprising a control step of performing control not to notify a determination result when the object protruding from the bed is determined to be only a futon in the determining step.

15. The monitoring method according to claim 13, further comprising a control step of performing control to notify a determination result when it is determined in the determination step that the object protruding from the bed includes at least a part of a body.

16. The monitoring method according to any one of claims 12 to 15, wherein the information obtained from the imaging unit is information obtained by analyzing an image captured by the imaging unit.

17. The monitoring method according to any one of claims 12 to 16, wherein the information obtained from the bed sensor is information obtained by analyzing the load detected by the bed sensor.

18. The monitoring method according to any one of claims 12 to 15 and 17, wherein the information obtained from the imaging unit is image information captured by the imaging unit.

19. The monitoring method according to any one of claims 12 to 16 and 18, wherein the information obtained from the bed sensor is load information detected by the bed sensor.

20. The monitoring method according to claim 18, further comprising the steps of: acquiring information from the bed sensor; and changing image recognition sensitivity based on the information obtained from the bed sensor, and then acquiring the image information.

21. The monitoring method according to any one of claims 12 to 20, further comprising an output step of outputting a determination result in said determining step.

22. A monitoring program for causing a computer to execute the monitoring method according to any one of claims 12 to 21.

Images