Monitoring system, monitoring device, and control method

Abstract

To provide a monitoring system, a monitoring device, and a control method with which it is possible to estimate an action of a person in need of care, even when trouble occurs to a detection unit.SOLUTION: The monitoring system comprises: a first detection unit and a second detection unit for detecting the state of a subject; a first estimation unit 112 for estimating an action of the subject on the basis of the state of the subject detected by at least the first detection unit out of the first detection unit and the second detection unit; an abnormality detection unit 113 for detecting an abnormal state that the first detection unit is incapable of detecting the state of the subject normally; and a second estimation unit 115 for estimating an action of the subject on the basis of the state of the subject detected by the second detection unit when the abnormal state is detected.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to a monitoring system, a monitoring device, and a control method.

Background Art

[0002] In Japan, due to the improvement of living standards, the improvement of sanitary environment, and the improvement of medical standards accompanying the high economic growth after the war, the aging of the population has become remarkable. Therefore, combined with the decline in the birth rate, it has become an aging society with a high aging rate. In such an aging society, an increase in the number of care recipients who require care, such as due to illness, injury, and aging, is assumed.

[0003] Care recipients may be injured due to falling during walking or falling from a bed in facilities such as hospitals and elderly welfare facilities, or at home. If caregivers such as nurses, nurses, facility staff, and family members can rush to the scene immediately according to the condition of the care recipient, it is possible to prevent the occurrence of injuries. For this reason, the development of a system for detecting the condition of a care recipient from an image captured by an imaging device has been underway (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In such a system, if a defect occurs in a detection unit such as an imaging device, there is a possibility that the behavior of a care recipient such as a fall cannot be estimated. Therefore, it is desirable to be able to estimate the behavior of a care recipient even when a defect occurs in the detection unit.

[0006] The present invention has been made to solve such problems. That is, an object of the present invention is to provide a monitoring system, a monitoring device, and a control method capable of estimating the behavior of a care recipient even when a malfunction occurs in a detection unit.

Means for Solving the Problems

[0007] The above problems of the present invention are solved by the following means.

[0008] (1) A first detection unit and a second detection unit for detecting the state of a target person, and a first estimation unit for estimating the behavior of the target person based on the state of the target person detected by at least the first detection unit among the first detection unit and the second detection unit, an abnormality detection unit for detecting an abnormal state in which the first detection unit cannot normally detect the state of the target person, and a second estimation unit for estimating the behavior of the target person based on the state of the target person detected by the second detection unit when the abnormal state is detected. A monitoring system comprising:

[0009] (2) The monitoring system according to (1) above, wherein the first detection unit includes a first imaging unit, and the abnormal state includes a shielding state in which the imaging field of the first imaging unit is shielded.

[0010] (3) The monitoring system according to (2) above, wherein the first imaging unit is provided on the bed of the target person.

[0011] (4) The monitoring system according to (2) above, wherein the first imaging unit is provided on the ceiling or wall of the room of the target person.

[0012] (5) The monitoring system according to (1) above, further comprising a notification unit for notifying estimation information regarding the behavior of the target person estimated by the first estimation unit or the second estimation unit.

[0013] (6) The monitoring system according to (5) above, wherein the notification unit further notifies detection information regarding the detection result by the abnormality detection unit.

[0014] (7) The monitoring system according to (5) above, further comprising a determination unit that determines whether or not to generate the estimated information, and when the determination unit determines to generate the estimated information, the notification unit notifies the estimated information.

[0015] (8) The monitoring system according to (1) above, wherein the first estimation unit estimates the behavior of the target person based on the state of the target person detected by the first detection unit and the second detection unit.

[0016] (9) The monitoring system according to (1) above, wherein the second detection unit includes a second imaging unit.

[0017] (10) The monitoring system according to (9) above, wherein the second imaging unit is provided on the bed of the target person.

[0018] (11) The monitoring system according to (1) above, wherein at least one of the first detection unit and the second detection unit includes a bed sensor provided on the bed of the target person.

[0019] (12) The monitoring system according to (2) above, further comprising a driving unit that moves the imaging field of view of the first imaging unit, and when the abnormal state is detected, the driving unit moves the imaging field of view of the first imaging unit.

[0020] (13) A monitoring device including a first detection unit and a second detection unit that detect the state of a target person, a first estimation unit that estimates the behavior of the target person based on the state of the target person detected by at least the first detection unit among the first detection unit and the second detection unit, an abnormality detection unit that detects an abnormal state in which the first detection unit cannot normally detect the state of the target person, a second estimation unit that estimates the behavior of the target person based on the state of the target person detected by the second detection unit when the abnormal state is detected, and an output unit that outputs estimated information regarding the behavior of the target person estimated by the first estimation unit or the second estimation unit.

[0021] A control method for a monitoring device having a first detection unit and a second detection unit for detecting the state of a target person, the method comprising: a step (a) of estimating the behavior of the target person based on the state of the target person detected by at least the first detection unit among the first detection unit and the second detection unit; a step (b) of detecting an abnormal state in which the first detection unit cannot normally detect the state of the target person; a step (c) of estimating the behavior of the target person based on the state of the target person detected by the second detection unit when the abnormal state is detected; and a step (d) of outputting estimation information regarding the behavior of the target person estimated by the step (a) or the step (c).

Advantages of the Invention

[0022] In the monitoring system, monitoring device, and control method according to the present invention, when an abnormal state of the first detection unit is detected, the behavior of the target person is estimated based on the state of the target person detected by the second detection unit. Therefore, even when a problem occurs in the detection unit, it is possible to estimate the behavior of the care recipient.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0024] Hereinafter, with reference to the drawings, a control device, a control method, a control program, and an information processing system according to an embodiment of the present invention will be described. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0025] (Embodiment) FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 according to an embodiment. The monitoring system 1 is used, for example, in a nursing care facility or the like. The monitoring system 1 includes a monitoring device 10, a server 20, a plurality of mobile terminals 30, and a wireless AP (access point) 31. The monitoring device 10, the server 20, and the mobile terminal 30 are communicably connected to each other via a LAN (Local Area Network) and the wireless AP 31. The monitoring device 10 is installed, for example, in the living room of the target person 80. The server 20 is managed by, for example, the administrator of the nursing care facility or the like. The mobile terminal 30 is carried by the caregiver 70 who takes care of or provides care for the target person 80. The target person 80 is a person in need of care such as nursing or care. Here, the mobile terminal 30 corresponds to a specific example of the notification unit.

[0026] (Monitoring device 10) FIG. 2 is a block diagram illustrating the hardware configuration of the monitoring device 10. The monitoring device 10 includes an arithmetic control unit 101, a communication unit 102, a first camera 1031, a second camera 1032, a third camera 1033, and a bed sensor 104. The arithmetic control unit 101, the communication unit 102, the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are interconnected by a bus. Here, any one of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 corresponds to a specific example of the first detection unit of the present invention, and the other corresponds to a specific example of the second detection unit. Any one of the first camera 1031, the second camera 1032, and the third camera 1033 corresponds to a specific example of the first imaging unit of the present invention, and the other corresponds to a specific example of the second imaging unit.

[0027] The arithmetic control unit 101 includes a CPU (Central Processing Unit), a RAM (Random-access memory), a ROM (Read Only Memory), etc. The arithmetic control unit 101 may have an HDD (hard disk drive) or an SSD (Solid State Drive). The arithmetic control unit 101 performs control and arithmetic processing of each part of the monitoring device 10 according to a control program. The main functions of the arithmetic control unit 101 will be described later.

[0028] The communication unit 102 is an interface circuit for communicating with, for example, a server 20 or the like via a LAN. The interface circuit is, for example, a LAN card and a wireless communication circuit, etc.

[0029] The first camera 1031, the second camera 1032, and the third camera 1033 image the monitoring area and output images or image data. The images captured by the first camera 1031, the second camera 1032, and the third camera 1033 include still images and moving images. The moving images may include a plurality of still images intermittently captured in time series. The first camera 1031, the second camera 1032, and the third camera 1033 are near-infrared cameras or visible light cameras. Some of the first camera 1031, the second camera 1032, and the third camera 1033 may be near-infrared cameras, and the rest may be visible light cameras. The first camera 1031, the second camera 1032, and the third camera 1033 are provided on the bed of the subject 80 (for example, the bed 90 in FIG. 3 described later).

[0030] FIG. 3 shows an example of the bed 90 of the subject 80. This bed 90 is provided in the living room of the subject 80. The bed 90 includes, for example, a frame portion 91, a headboard 92, a footboard 93, and a mattress 94. The bed 90 is, for example, an electric bed. For example, by tilting the mattress 94 at a portion corresponding to the back of the subject 80, a tilted (reclining) state can be created. The bed 90 has, for example, a rectangular planar shape.

[0031] The first camera 1031, the second camera 1032, and the third camera 1033 are, for example, attached to the frame portion 91. The first camera 1031, the second camera 1032, and the third camera 1033 are provided at different positions on the bed 90. The first camera 1031 has an imaging field of view F1, the second camera 1032 has an imaging field of view F2, and the third camera 1033 has an imaging field of view F3. It is preferable that the imaging field of view F1, the imaging field of view F2, and the imaging field of view F3 do not overlap with each other. In other words, it is preferable that the first camera 1031, the second camera 1032, and the third camera 1033 image different regions from each other. This makes it possible to image the subject 80 from multiple angles.

[0032] For example, the imaging field of view F2 of the second camera 1032 includes the area on the bed 90. For example, the imaging field of view F1 of the first camera 1031 includes one area outside the bed 90. For example, the imaging field of view F3 of the third camera 1033 includes the other area outside the bed 90.

[0033] The first camera 1031, the second camera 1032, and the third camera 1033 are provided, for example, along one side of the bed 90. Thereby, the settings of each of the first camera 1031, the second camera 1032, and the third camera 1033 can be made closer to each other. Therefore, image processing of the images captured by each of the first camera 1031, the second camera 1032, and the third camera 1033 becomes easy. The first camera 1031, the second camera 1032, and the third camera 1033 are provided, for example, along the short side of the bed 90. The first camera 1031, the second camera 1032, and the third camera 1033 are provided, for example, along the short side on the footboard 93 side.

[0034] Figures 4 and 5 show other examples of the installation positions of the first camera 1031, the second camera 1032, and the third camera 1033. The first camera 1031, the second camera 1032, and the third camera 1033 may be provided along the short side on the headboard 92 side (Figure 4). Alternatively, the first camera 1031, the second camera 1032, and the third camera 1033 may be provided across a plurality of sides of the bed 90 (Figure 5). For example, the first camera 1031 and the third camera 1033 may be provided on each of the opposing long sides of the bed 90, and the second camera 1032 may be provided on the short side of the bed 90. For example, a part of each of the imaging field of view F1, the imaging field of view F2, and the imaging field of view F3 may overlap. For example, a part of the imaging field of view F1 and a part of the imaging field of view F3 overlap, and a part of the imaging field of view F2 and a part of the imaging field of view F3 overlap (Figures 4 and 5).

[0035] FIG. 6 shows an example of an image Im1 captured by the first camera 1031, an image Im2 captured by the second camera 1032, and an image Im3 captured by the third camera 1033. For example, the feet of the subject 80 outside the bed 90 are shown in the image Im1. For example, the subject 80 outside the bed 90 from one end of the bed 90 is shown in the image Im3. For example, the subject 80 in the center on the bed 90 is shown in the image Im2.

[0036] FIG. 7 shows another example of the image Im1. The subject 80 outside the bed 90 from the other end of the bed 90 may be shown in the image Im1. The feet of the subject 80 outside the bed 90 may be shown in the image Im3 (not shown).

[0037] Part or all of the first camera 1031, the second camera 1032, and the third camera 1033 may be provided on the ceiling or wall of the room of the subject 80.

[0038] The bed sensor 104 is disposed on the bed 90 in the room of the subject 80 (FIG. 3). The bed sensor 104 outputs, for example, a signal corresponding to the load on the bed 90. The bed sensor 104 is composed of, for example, a sheet-like bag filled with air and a piezoelectric sensor that detects fluctuations in the pressure of this air.

[0039] The bed sensor 104 may detect the body movement of the subject 80. This bed sensor 104 detects, for example, the Doppler shift of microwaves generated by the body movement (e.g., breathing movement) of the subject 80 by transmitting microwaves to a predetermined irradiation area and receiving the reflected waves. The bed sensor 104 may detect the sleep, waking, getting up, getting out of bed, staying in bed, respiratory rate, and heart rate of the subject 80.

[0040] (Server 20) FIG. 8 is a block diagram showing the hardware configuration of the server 20. The server 20 includes an arithmetic control unit 201, a communication unit 202, a storage unit 203, and a display unit 204. The arithmetic control unit 201, the communication unit 202, the storage unit 203, and the display unit 204 are interconnected by a bus.

[0041] The arithmetic control unit 201 includes a CPU, a RAM, a ROM, etc. The arithmetic control unit 201 controls and performs arithmetic processing on each part of the server 20 according to a program.

[0042] The communication unit 202 is constituted by an interface circuit for communicating with, for example, a mobile terminal 30 via a LAN.

[0043] The storage unit 203 is constituted by an HDD or an SSD, and stores various programs and various data.

[0044] The display unit 204 displays various information.

[0045] (Mobile terminal 30) FIG. 9 is a block diagram showing the hardware configuration of the mobile terminal 30. The mobile terminal 30 includes a control unit 301, a wireless communication unit 302, a display unit 303, an input unit 304, and an audio input / output unit 305. The control unit 301, the wireless communication unit 302, the display unit 303, the input unit 304, and the audio input / output unit 305 are interconnected by a bus. The mobile terminal 30 can be constituted by a portable communication terminal device such as, for example, a tablet computer, a smartphone, or a mobile phone.

[0046] The control unit 301 includes a CPU, a RAM, a ROM, etc. The control unit 301 controls and performs arithmetic processing on each part of the mobile terminal 30 according to a program.

[0047] The wireless communication unit 302 performs wireless communication with each device via the wireless AP 31 or directly using standards such as Wi-Fi and Bluetooth (registered trademark). In the mobile terminal 30, in addition to the wireless LAN settings, access from the mobile terminal 30 to the server 20 may be enabled by inputting connection information with the server 20 (the IP address of the server 20 and the port number set in the server 20).

[0048] The display unit 303 and the input unit 304 are configured by, for example, a touch panel in which a touch sensor is superimposed on a display surface such as a liquid crystal. The display unit 303 and the input unit 304 display various information and accept various inputs.

[0049] The voice input / output unit 305 is, for example, a speaker and a microphone, and enables voice calls by the caregiver 70 between the mobile terminal 30 and other mobile terminals 30 via the wireless communication unit 302.

[0050] (Functions of the monitoring device 10) FIG. 10 is a block diagram illustrating the main functions of the arithmetic control unit 101 of the monitoring device 10. The arithmetic control unit 101 functions as, for example, a state information acquisition unit 111, a first estimation unit 112, an abnormality detection unit 113, a second estimation unit 114, a determination unit 115, and an output unit 116.

[0051] The state information acquisition unit 111 acquires state information regarding the state of the subject 80. The state information is information regarding the state of the subject 80 detected by each of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104. The state information includes, for example, image information regarding an image captured by each of the first camera 1031, the second camera 1032, and the third camera 1033, and load information regarding the load on the bed 90 detected by the bed sensor 104.

[0052] The first estimation unit 112 estimates the actions of the target person 80 based on the state information acquired by the state information acquisition unit 111. The first estimation unit 112 estimates the actions of the target person 80 based on the state information detected by at least any one of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104. At this time, the first estimation unit 112 estimates the actions of the target person 80 using the first method.

[0053] The first estimation unit 112 estimates the actions of the target person 80, for example, by performing image analysis on the images captured by each of the first camera 1031, the second camera 1032, and the third camera 1033. The actions of the target person 80 are, for example, getting up, getting into bed, getting out of bed, lying in bed, falling, transferring, and falling off the bed, etc. By using the state information detected by each of the plurality of detection units (the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104), it is possible to estimate the actions of the target person 80 with higher accuracy compared to the case of using a single detection unit.

[0054] For example, when the target person 80 recognized in the image Im2 captured by the second camera 1032 is no longer recognized in the image Im2 thereafter, the first estimation unit 112 estimates that the target person 80 has moved outside the bed 90. The first estimation unit 112 recognizes the target person 80 from the images Im1, Im2, and Im3 using, for example, the frame difference method or a learned model, etc. The movement of the target person 80 outside the bed 90 is, for example, getting up, getting out of bed, and falling off the bed, etc.

[0055] The first estimation unit 112 may further estimate the actions of the subject 80 by combining at least one of the image Im1 captured by the first camera 1031 and the image Im3 captured by the third camera 1033. For example, further, when the feet of the subject 80 are recognized in the image Im1, the first estimation unit 112 estimates the movement of the subject 80 out of the bed 90. For example, further, when the subject 80 is recognized at one end of the bed 90 or outside the bed 90 in the image Im3, the first estimation unit 112 estimates the movement of the subject 80 out of the bed 90.

[0056] For example, when the subject 80 who was not recognized in the image Im2 captured by the second camera 1032 is then recognized in the image Im2, the first estimation unit 112 estimates the movement of the subject 80 into the bed 90. The movement of the subject 80 into the bed 90 is, for example, getting into bed and lying down. The first estimation unit 112 may further estimate the actions of the subject 80 by combining at least one of the image Im1 captured by the first camera 1031 and the image Im3 captured by the third camera 1033. For example, further, when the feet of the recognized subject 80 are no longer recognized in the image Im1, the first estimation unit 112 estimates the movement of the subject 80 into the bed 90. For example, further, when the subject 80 who was recognized at one end of the bed 90 or outside the bed 90 is no longer recognized in the image Im3, the first estimation unit 112 estimates the movement of the subject 80 into the bed 90.

[0057] The first estimation unit 112 may estimate the actions of the subject 80 using a learned model. The first estimation unit 112 may estimate the actions of the subject 80 using the state information detected by the bed sensor 104. The first estimation unit 112 may estimate the actions of the subject 80 by combining at least one of the first camera 1031, the second camera 1032, and the third camera 1033 with the bed sensor 104.

[0058] The abnormality detection unit 113 detects an abnormal state. The abnormal state is, for example, a state in which at least one of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 cannot normally detect the state of the subject. The abnormal state includes, for example, a shielding state in which at least one of the imaging fields of view F1 of the first camera 1031, F2 of the second camera 1032, and F3 of the third camera 1033 is shielded. The shielding state is, for example, a state in which the imaging fields of view F1, F2, and F3 are covered with an object or the like so that the behavior of the subject 80 cannot be estimated from the captured image. The shielding state includes a state in which a part of each of the imaging fields of view F1, F2, and F3 is shielded. The shielding state is, for example, a state in which at least one lens portion of the first camera 1031, the second camera 1032, and the third camera 1033 is covered with an object in the living room of the subject 80. The objects in the living room of the subject 80 are, for example, futons, clothes, movable tables, desks, wheelchairs, portable toilets, and furniture.

[0059] The abnormality detection unit 113 detects the shielding state, for example, based on the images captured by the first camera 1031, the second camera 1032, and the third camera 1033. The abnormality detection unit 113 can detect the shielding state using, for example, known image processing techniques or image recognition techniques.

[0060] The abnormality detection unit 113 detects the shielding state, for example, based on the number of pixels of the segmentation data such as people, futons, or beds in the image. For example, when 60% or more of the number of pixels in the image are occupied by the segmentation data of a futon or a bed, the abnormality detection unit 113 detects the shielding state.

[0061] The abnormality detection unit 113 may detect the shielding state by comparing the images over time. For example, when an object such as a person, a futon, or furniture is recognized in the image at the first time, and this object recognition is not performed in the image at the second time after the first time, the abnormality detection unit 113 detects the shielding state.

[0062] The abnormality detection unit 113 may detect a shielding state based on the pixel values of each pixel constituting the image. The abnormality detection unit 113 may detect the shielding state based on, for example, the histogram or standard deviation of the pixel values of each pixel. For example, when the luminance of a predetermined ratio or more of the pixels constituting the image is equal to or less than a threshold value, the abnormality detection unit 113 detects a shielding state. The abnormality detection unit 113 may detect the shielding state using the number of pixels from which edges are extracted from the image. For example, when the number of pixels from which edges are extracted is equal to or less than a threshold value, the abnormality detection unit 113 detects a shielding state.

[0063] The abnormality detection unit 113 may detect the shielding state using the background difference method. The abnormality detection unit 113 may detect the shielding state when, for example, there is an abnormality in the difference between the background image and the image at a predetermined time. For example, when there is almost no difference between the background image and the images at each time over a predetermined period, the abnormality detection unit 113 detects a shielding state.

[0064] The abnormality detection unit 113 may detect the shielding state using the luminance change detection method. The abnormality detection unit 113 may detect the shielding state when, for example, there is a region having a rapid luminance change in a plurality of images captured over time.

[0065] The abnormality detection unit 113 may detect the shielding state using the motion detection method. The abnormality detection unit 113 may detect the shielding state when, for example, the motion pattern of an object detected from the image deviates from the normal motion pattern.

[0066] The abnormality detection unit 113 may detect the shielding state using, for example, deep learning-based segmentation. The abnormality detection unit 113 may segment the shielded region in the image using a deep learning model such as Mask R-CNN. At this time, the abnormality detection unit 113 detects the shielding state.

[0067] The abnormality detection unit 113 may detect the occlusion state using texture analysis, transparency analysis, etc. The abnormality detection unit 113 may detect the occlusion state using a distance image. For example, when the number of pixels having a distance equal to or less than a predetermined value is equal to or greater than a threshold value, the abnormality detection unit 113 detects the occlusion state.

[0068] The abnormal state includes, for example, a state in which the viewing angle of at least one of the first camera 1031, the second camera 1032, and the third camera 1033 has changed. For example, when the target person 80 is not recognized in the image Im2 over a predetermined period, the abnormality detection unit 113 detects that the viewing angle of the second camera 1032 has changed.

[0069] The abnormal state includes, for example, a state in which at least one of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 does not operate normally. For example, when there is no data output from the bed sensor 104 over a predetermined period, the abnormality detection unit 113 detects that the bed sensor 104 does not operate normally.

[0070] The abnormality detection unit 113 may detect an abnormal state using a machine learning model. Data when the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are operating normally and data of the abnormal state are input to the machine learning model, for example. The abnormality detection unit 113 may detect that two or more of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are in an abnormal state.

[0071] When the abnormality detection unit 113 detects an abnormal state, the second estimation unit 114 estimates the behavior of the subject 80. The second estimation unit 114 estimates the behavior of the subject 80 using detection units other than the detection unit in which the abnormal state is detected by the abnormality detection unit 113. At this time, the second estimation unit 114 estimates the behavior of the subject 80 using a second method. The second method is different from the first method used by the first estimation unit 112. In the present embodiment, since the monitoring device 10 includes this second estimation unit 114, even when a malfunction occurs in the first camera 1031, the second camera 1032, the third camera 1033, or the bed sensor 104, the behavior of the subject 80 can be estimated.

[0072] For example, the parameters used by the second estimation unit 114 are different from the parameters used by the first estimation unit 112. For example, the threshold value used by the second estimation unit 114 is different from the threshold value used by the first estimation unit 112. For example, in the second estimation unit 114, the time required to estimate a predetermined behavior of the subject 80 is longer than that in the first estimation unit 112. Thereby, the error of estimation is reduced, and it becomes possible to accurately determine important behaviors of the subject 80 such as falling. Alternatively, in the second estimation unit 114, the time required to estimate a predetermined behavior of the subject 80 is shorter than that in the first estimation unit 112. Thereby, it is possible to reduce the omission of estimation of the behavior of the subject 80 and reduce the overlooking of the behavior of the subject 80.

[0073] For example, in the second estimation unit 114, the sensitivity for estimating a predetermined behavior of the subject 80 is lower than that in the first estimation unit 112. Thereby, the error of estimation is reduced, and it becomes possible to accurately determine important behaviors of the subject 80 such as falling. Alternatively, in the second estimation unit 114, the sensitivity for estimating a predetermined behavior of the subject 80 is higher than that in the first estimation unit 112. Thereby, it is possible to reduce the omission of estimation of the behavior of the subject 80 and reduce the overlooking of the behavior of the subject 80.

[0074] For example, when the abnormality detection unit 113 detects an abnormal state of the second camera 1032, the second estimation unit 114 uses the state information detected by at least one of the first camera 1031, the third camera 1033, and the bed sensor 104. For example, when the feet of the subject 80 are recognized in the image Im1 captured by the first camera, the second estimation unit 114 estimates the movement of the subject 80 out of the bed 90. Alternatively, when the subject 80 is recognized at one end of the bed 90 or outside the bed 90 in the image Im3 captured by the third camera 1033, the second estimation unit 114 estimates the movement of the subject 80 out of the bed 90. Alternatively, when it is recognized that the load on the bed 90 has decreased based on the load information detected by the bed sensor 104, the second estimation unit 114 estimates the movement of the subject 80 out of the bed 90. The second estimation unit 114 may estimate the behavior of the subject 80 using the state information detected by the first camera 1031, the third camera 1033, and the bed sensor 104.

[0075] For example, when the abnormality detection unit 113 detects an abnormal state of the first camera 1031, the second estimation unit 114 uses the state information detected by at least one of the second camera 1032, the third camera 1033, and the bed sensor 104. For example, when the abnormality detection unit 113 detects an abnormal state of the third camera 1033, the second estimation unit 114 uses the state information detected by at least one of the first camera 1031, the second camera 1032, and the bed sensor 104. For example, when the abnormality detection unit 113 detects an abnormal state of the bed sensor 104, the second estimation unit 114 uses the state information detected by at least one of the first camera 1031, the second camera 1032, and the third camera 1033.

[0076] The determination unit 115 determines whether to generate the estimation information. The estimation information is information regarding the actions of the target person 80 estimated by the first estimation unit 112 or the second estimation unit 114. The estimation information includes, for example, the identification information of the target person 80, information regarding the type of the estimated action, information regarding the estimation time, and the like. By including this determination unit 115, the monitoring device 10 can reduce unnecessary notifications to the caregiver 70.

[0077] For example, based on the type of the action of the target person 80 estimated by the first estimation unit 112 or the second estimation unit 114, the determination unit 115 determines whether to generate the estimation information. For example, when it is estimated that the target person 80 has fallen, the determination unit 115 determines to generate the estimation information. For example, after it is estimated that the target person 80 has woken up, the determination unit 115 determines to generate all the estimation information. Thereby, it becomes easier for the caregiver 70 to notice the fall or the like of the target person 80. While the target person 80 is estimated to be in the bed 90, the determination unit 115 may determine to generate all the estimation information.

[0078] The determination unit 115 may determine whether to generate the estimation information based on the setting state of the notification switch. For example, when the notification switch is in the on state, the determination unit 115 determines to generate the estimation information. The notification switch is provided, for example, in any one of the monitoring device 10, the server 20, and the mobile terminal 30.

[0079] The determination unit 115 may determine whether to generate the estimation information based on the number of people present in the room of the target person 80. For example, when the number of people present in the room of the target person 80 is one, the determination unit 115 determines to generate the estimation information. This is because when there are a plurality of people present in the room of the target person 80, the possibility of overlooking the fall or the like of the target person 80 is low.

[0080] The determination unit 115 may determine whether to generate the estimation information based on the attributes of the people present in the room of the target person 80. For example, when the person present in the room of the target person 80 is the caregiver 70, the determination unit 115 determines not to generate the estimation information.

[0081] The determination unit 115 may determine whether to generate estimation information based on an image captured by at least one of the first camera 1031, the second camera 1032, and the third camera 1033. For example, when an abnormal state of the first camera 1031 is detected, the determination unit 115 determines whether to generate estimation information based on an image captured by at least one of the second camera 1032 and the third camera 1033. For example, when an image captured by the second camera 1032 or the third camera 1033 shows a plurality of people, the determination unit 115 determines not to generate the estimation information.

[0082] The determination unit 115 may further determine whether to generate detection information. The detection information is information regarding the detection result by the abnormality detection unit 113. The detection information includes, for example, information regarding the detection time by the abnormality detection unit 113, information regarding the imaging time of the image used by the abnormality detection unit 113, and information regarding whether an abnormal state has been detected by the abnormality detection unit 113.

[0083] When an abnormal state is detected by the abnormality detection unit 113, the detection information includes the identification information of the detection unit in which the abnormal state has been detected. For example, when an abnormal state of the first camera 1031 is detected, the detection information includes the first identification information of the first camera 1031. When an abnormal state of the second camera 1032 is detected, the detection information includes the second identification information of the second camera 1032. When an abnormal state of the third camera 1033 is detected, the detection information includes the third identification information of the third camera 1033. When an abnormal state of the bed sensor 104 is detected, the detection information includes the fourth identification information of the bed sensor 104. When abnormal states of a plurality of detection units are detected, the detection information includes the identification information of each of the plurality of detection units.

[0084] For example, when the abnormality detection unit 113 detects an abnormal state, the determination unit 115 determines that detection information is to be generated. For example, when an abnormal state has been continuously detected for a predetermined period of time, the determination unit 115 determines that detection information is to be generated. For example, when an abnormal state has been continuously detected for 20 seconds or more, the determination unit 115 determines that detection information is to be generated.

[0085] The determination unit 115 may determine whether to generate detection information in accordance with the determination of whether to generate estimation information. For example, when the determination unit 115 determines to generate estimation information, it determines to generate detection information.

[0086] When it is determined by the determination unit 115 that estimation information is to be generated, the output unit 116 generates and outputs the estimation information. When it is determined by the determination unit 115 that detection information is to be generated, the output unit 116 generates and outputs the detection information. For example, the output unit 116 outputs the estimation information by transmitting the estimation information to the mobile terminal 30 using the communication unit 102. The mobile terminal 30 that has received the estimation information causes, for example, the display unit 303 to display information indicating the actions of the target person 80. The output unit 116 outputs the detection information by transmitting the detection information to the mobile terminal 30 using the communication unit 102, for example.

[0087] The mobile terminal 30 that has received the estimation information or the detection information causes, for example, the display unit 303 to display a predetermined icon and causes the audio input / output unit 305 to output a predetermined sound. Thereby, the caregiver 70 can be alerted to a predetermined action of the target person 80. Alternatively, the caregiver 70 can be alerted to an abnormality in the first camera 1031, the second camera 1032, the third camera 1033, or the bed sensor 104. The output unit 116 may adjust the timing to output the estimation information and the detection information so that the mobile terminal 30 simultaneously notifies the estimation information and the detection information.

[0088] In the display unit 303, for example, information indicating the first identification information, the second identification information, the third identification information, and the fourth identification information included in the detection information is further displayed. As a result, the caregiver 70 can immediately grasp which of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 has an abnormality.

[0089] The output unit 116 may output the estimation information and the detection information by transmitting the estimation information and the detection information to the server 20 using the communication unit 102. The server 20 that has received the estimation information or the detection information causes, for example, a predetermined icon to be displayed on the display unit 204. The output unit 116 may output the estimation information and the detection information by transmitting the estimation information and the detection information to an external device using the communication unit 102. For example, the output unit 116 may transmit the estimation information and the detection information to a lamp or a speaker installed outside the living room of the subject 80.

[0090] (Control method of the monitoring device 10) FIG. 11 is a flowchart illustrating a processing procedure of the control method of the monitoring device 10. The processing of the flowchart in the figure is realized by the CPU of the arithmetic control unit 101 executing a control program.

[0091] First, the monitoring device 10 acquires state information (step S101). The state information is information regarding the state of the subject 80 detected by each of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 as described above.

[0092] Next, the monitoring device 10 determines whether an abnormal state has been detected (step S102). When it is determined that an abnormal state has not been detected (step S102: NO), the monitoring device 10 estimates the behavior of the subject 80 using the first method by the first estimation unit 112 (step S103). At this time, the monitoring device 10 estimates the behavior of the subject 80 based on the state information acquired in step S101.

[0093] When it is determined that an abnormal state has been detected (step S102: YES), the monitoring device 10 estimates the behavior of the subject 80 using the second method by the second estimation unit 114 (step S104). At this time, the monitoring device 10 estimates the behavior of the subject 80 based on the state information detected by the detection units other than the detection unit in which the abnormal state has been detected. For example, when an abnormal state of the second camera 1032 is detected, the monitoring device 10 estimates the behavior of the subject 80 based on the state information detected by at least one of the first camera 1031, the third camera 1033, and the bed sensor 104.

[0094] After performing the process of step S103 or step S104, the monitoring device 10 determines whether to generate estimation information (step S105). When it is determined not to generate the estimation information (step S105: NO), the monitoring device 10 returns to the process of step S101. When it is determined to generate the estimation information (step S105: YES), the monitoring device 10 generates and outputs the estimation information (step S106) and ends the process.

[0095] (Functions and Effects of the Monitoring Device 10 and the Monitoring System 1) In the monitoring device 10 and the monitoring system 1 of the present embodiment, abnormal states of each of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are detected. Further, in this monitoring device 10 and the monitoring system 1, when an abnormal state is detected, the behavior of the subject 80 is estimated based on the state of the subject 80 detected by other detection units. Therefore, even when a problem occurs in some of the plurality of detection units, it is possible to estimate the behavior of the subject 80. Hereinafter, this function and effect will be described.

[0096] Compared with the case where one camera is installed in the living room of the subject, by installing a plurality of cameras, it is possible to estimate the behavior of the subject with higher accuracy. In particular, by providing a plurality of cameras around the bed, it becomes easier for the caregiver to notice events such as the subject getting up and getting out of bed, which require the assistance of the caregiver.

[0097] However, there are various items such as futons, clothes, and movable tables on the bed. Also, items such as wheelchairs and furniture may be placed around the bed. If these items cover the imaging field of the camera, the camera cannot capture an appropriate image of the target person, and it becomes difficult to estimate the actions of the target person from the image. For this reason, there is a risk that the caregiver may not notice the target person's fall or the like.

[0098] On the other hand, in the monitoring device 10, when an abnormality occurs in any of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104, the actions of the target person 80 are estimated using a detection unit different from the detection unit in which the abnormality has occurred. For example, when the imaging field of the second camera 1032 is covered by an object or the like, the actions of the target person 80 are estimated using at least one of the first camera 1031, the third camera 1033, and the bed sensor 104. Therefore, even when a malfunction occurs in the second camera 1032, the caregiver 70 or the like can grasp the actions of the target person 80 and perform appropriate care for the target person 80.

[0099] In addition, the monitoring device 10 can output detection information regarding the detection result by the abnormality detection unit 113. Thereby, the caregiver 70 or the like can notice that an abnormality has occurred in the first camera 1031, the second camera 1032, the third camera 1033, or the bed sensor 104, and for example, it becomes possible to move the shielding object.

[0100] In addition, in the monitoring device 10, it is determined whether or not to generate estimation information. For example, if all the estimation information is notified to the caregiver 70 or the like, there is a risk of burdening the caregiver 70 or the like with unnecessary work. By the monitoring device 10 determining whether or not to generate the estimation information, it becomes possible to reduce unnecessary notifications to the caregiver 70 or the like.

[0101] As described above, in the monitoring device 10 and the monitoring system 1 of the present embodiment, abnormal states of each of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are detected. At this time, in the monitoring device 10 and the monitoring system 1, the behavior of the subject 80 is estimated based on the state of the subject 80 detected by other detection units. Therefore, even when an abnormality occurs in a part of the plurality of detection units, it is possible to estimate the behavior of the subject 80.

[0102] Hereinafter, a modified example of the monitoring device 10 described in the above embodiment will be described. To avoid duplication of explanation, detailed explanation of the same configuration as that of the above embodiment will be omitted.

[0103] (Modified Example 1) In the monitoring device 10, the first estimation unit 112 may estimate the behavior of the subject 80 using the first algorithm, and the second estimation unit 114 may estimate the behavior of the subject 80 using the second algorithm. The second algorithm is an algorithm different from the first algorithm.

[0104] FIG. 12 shows an example of an image Im1 captured by the first camera 1031 and an image Im3 captured by the third camera 1033. The first estimation unit 112 estimates the behavior of the subject 80 using, for example, the determination region A1 in the image Im1 and the determination region A3 in the image Im3. For example, when the subject 80 is recognized at the end portion A11 of the determination region A1 and the subject 80 is recognized in the determination region A3, the first estimation unit 112 estimates that the subject 80 moves outside the bed 90.

[0105] When an abnormal state of the first camera 1031 is detected, the second estimation unit 114 estimates the behavior of the subject 80 using, for example, the determination area A3 in the image Im3. For example, when the subject 80 is recognized in the determination area A3, the second estimation unit 114 estimates the movement of the subject 80 out of the bed 90. When an abnormal state of the third camera 1033 is detected, the second estimation unit 114 estimates the behavior of the subject 80 using, for example, the determination area A1 in the image Im1. For example, when the subject 80 is recognized at the end portion A11 of the determination area A1, the second estimation unit 114 estimates the movement of the subject 80 out of the bed 90. Thus, the first estimation unit 112 and the second estimation unit 114 may estimate the behavior of the subject 80 using different algorithms from each other.

[0106] In the monitoring device 10 according to this modification 1, similar to that described in the above embodiment, abnormal states of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are each detected. At this time, in the monitoring device 10 and the monitoring system 1, the behavior of the subject 80 is estimated based on the state of the subject 80 detected by another detection unit. Therefore, even when an abnormality occurs in a part of the plurality of detection units, it is possible to estimate the behavior of the subject 80.

[0107] (Modification 2) FIG. 13 is a block diagram illustrating the hardware configuration of the monitoring device 10 according to Modification 2. The monitoring device 10 may include a drive unit 105 in addition to the arithmetic control unit 101, the communication unit 102, the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104.

[0108] The drive unit 105 changes the imaging field of view of at least any one of the first camera 1031, the second camera 1032, and the third camera 1033 (for example, the imaging fields of view F1, F2, and F3 in FIG. 3). The drive unit 105 includes, for example, a motor and a gear. The first camera 1031, the second camera 1032, and the third camera 1033 rotate along a predetermined axis, and the imaging field of view is changed.

[0109] When the arithmetic control unit 101 detects an abnormal state of the first camera 1031, for example, it changes the imaging field of view of the first camera 1031 by the driving unit 105. The arithmetic control unit 101 changes the imaging field of view of the first camera 1031 until, for example, the abnormal state of the first camera 1031 is no longer detected. Thereby, when the first camera 1031 becomes in an abnormal state due to shielding by an object, a change in the angle of view, etc., it is possible to return the first camera 1031 to a state where it can be normally imaged without bothering the caregiver 70. When the arithmetic control unit 101 detects an abnormal state of the second camera 1032, it may change the imaging field of view of the second camera 1032 by the driving unit 105. When the arithmetic control unit 101 detects an abnormal state of the third camera 1033, it may change the imaging field of view of the third camera 1033 by the driving unit 105.

[0110] In the monitoring device 10 according to this modification 2, as in the above-described embodiment, abnormal states of the first camera 1031, the second camera 1032, the third camera 1033, and the bed sensor 104 are each detected. At this time, in the monitoring device 10 and the monitoring system 1, the behavior of the subject 80 is estimated based on the state of the subject 80 detected by other detection units. Therefore, even when an abnormality occurs in a part of the plurality of detection units, it is possible to estimate the behavior of the subject 80.

[0111] Further, since the monitoring device 10 has the driving unit 105, when an abnormal state of the first camera 1031, the second camera 1032, or the third camera 1033 is detected, it is possible to return them to a state where they can be normally imaged without bothering the caregiver 70.

[0112] The configurations of the monitoring device 10 and the monitoring system 1 described above explain the main configurations in describing the features of the above-described embodiment and modification, and are not limited to the above-described configurations, and can be variously modified within the scope of the claims.

[0113] For example, the monitoring device 10 may include at least two detection units, or may include four or more detection units.

[0114] Also, the installation positions of the first camera 1031, the second camera 1032, and the third camera 1033 may be other positions. For example, the first camera 1031 and the second camera 1032 may be provided on the bed 90, and the third camera 1033 may be provided on the ceiling or wall of the living room of the subject 80.

[0115] Also, in the above-described embodiment, an example of determining whether the monitoring device 10 generates estimation information and detection information has been described, but the monitoring device 10 may always generate estimation information and detection information. The monitoring device 10 may generate detection information, for example, when no abnormal state is detected.

[0116] Also, the bed 90 may be configured to tilt the mattress 94 and the frame portion 91 not only electrically but also by hydraulic pressure or the like. The bed 90 may be configured such that the mattress 94 and the frame portion 91 are fixed.

[0117] Also, the monitoring device 10 may be installed in a place other than a nursing care facility, for example, in the home of the subject 80.

[0118] Also, some of the functions of the monitoring device 10 described in the above embodiment and the modification example may be possessed by other devices such as the server 20. That is, the monitoring device 10 may constitute the monitoring device of the present invention together with other devices.

[0119] Also, in the above-described flowchart, steps other than the steps shown in the flowchart may be included, or some steps may not be included. Also, the order of the steps is not limited to the above-described embodiment. Further, each step may be executed as one step in combination with other steps, may be executed included in other steps, or may be divided into a plurality of steps and executed.

[0120] In addition, the means and methods for performing various processes in the above-described embodiments can be realized by either dedicated hardware circuits or programmed computers. The above program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a storage unit such as a hard disk. Further, the above program may be provided as a single application software, or may be incorporated as a single function into the software of a device such as its detection unit.

Description of Reference Numerals

[0121] 1 Monitoring system, 10 Monitoring device, 101 Arithmetic control unit, 102 Communication unit, 1031 First camera, 1032 Second camera, 1033 Third camera, 104 Bed sensor, 105 Driving unit, 111 State information acquisition unit, 112 First estimation unit, 113 Abnormality detection unit, 114 Second estimation unit, 115 Judgment unit, 116 Output unit, 20 Server, 201 Arithmetic control unit, 202 Communication unit, 203 Storage unit, 204 Display unit, 30 Mobile terminal, 301 Control unit, 302 Wireless communication unit, 303 Display unit, 304 Input unit, 305 Audio input / output unit, 31 access points, 90 beds, 91 frame parts, 92 headboards, 93 footboards, 94 mattresses.

Claims

1. A first detection unit and a second detection unit that detect the state of a target person, Based on the state of the target person detected by at least the first detection unit among the first detection unit and the second detection unit, a first estimation unit that estimates the behavior of the target person, An abnormality detection unit that detects an abnormal state in which the first detection unit cannot normally detect the state of the target person, When the abnormal state is detected, a second estimation unit that estimates the behavior of the target person based on the state of the target person detected by the second detection unit A monitoring system comprising.

2. The first detection unit includes a first imaging unit, The monitoring system according to claim 1, wherein the abnormal state includes a shielding state in which the imaging field of view of the first imaging unit is shielded.

3. The monitoring system according to claim 2, wherein the first imaging unit is provided on the bed of the target person.

4. The monitoring system according to claim 2, wherein the first imaging unit is provided on the ceiling or wall of the living room of the target person.

5. The monitoring system according to claim 1, further comprising a notification unit that notifies estimation information regarding the behavior of the target person estimated by the first estimation unit or the second estimation unit.

6. The monitoring system according to claim 5, wherein the notification unit further notifies detection information regarding the detection result by the abnormality detection unit.

7. Further comprising a determination unit that determines whether to generate the estimation information, When the determination unit determines that the estimation information is to be generated, the notification unit notifies the estimation information according to claim 5.

8. The monitoring system according to claim 1, wherein the first estimation unit estimates the behavior of the target person based on the states of the target person detected by the first detection unit and the second detection unit.

9. The monitoring system according to claim 1, wherein the second detection unit includes a second imaging unit.

10. The monitoring system according to claim 9, wherein the second imaging unit is provided on the bed of the target person.

11. The monitoring system according to claim 1, wherein at least one of the first detection unit and the second detection unit includes a bed sensor provided on the bed of the target person.

12. Further comprising a driving unit that moves the imaging field of view of the first imaging unit, When the abnormal state is detected, the driving unit moves the imaging field of view of the first imaging unit according to claim 2.

13. A first detection unit and a second detection unit that detect the state of a target person, A first estimation unit that estimates the behavior of the subject based on the state of the subject detected by at least the first detection unit among the first detection unit and the second detection unit; An abnormality detection unit that detects an abnormal state in which the first detection unit cannot normally detect the state of the subject; A second estimation unit that estimates the behavior of the subject based on the state of the subject detected by the second detection unit when the abnormal state is detected; An output unit that outputs estimation information regarding the behavior of the subject estimated by the first estimation unit or the second estimation unit A monitoring device comprising the above.

14. A control method for a monitoring device having a first detection unit and a second detection unit that detect the state of a subject, the method comprising: Step (a) of estimating the behavior of the subject based on the state of the subject detected by at least the first detection unit among the first detection unit and the second detection unit; Step (b) of detecting an abnormal state in which the first detection unit cannot normally detect the state of the subject; Step (c) of estimating the behavior of the subject based on the state of the subject detected by the second detection unit when the abnormal state is detected; Step (d) of outputting estimation information regarding the behavior of the subject estimated by step (a) or step (c) A control method including the above.

Images