Watching system, watching method, and watching program

Abstract

To provide a watching system capable of easily reducing the risk of over-turning of a care-needing person, etc., and detecting a subtle change in behavior of the care-needing person, etc.SOLUTION: A watching system includes: an acquisition section for acquiring an image obtained by imaging a predetermined region; an object state detection section for detecting a state of an object used by a target person in the acquired image; a calculation section for calculating a normal state of the object based on past states of the object; a change detection section for detecting a change in the detected state of the object with respect to the calculated normal state; and a notification section for performing a notification of an alarm when the change is detected.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to a monitoring system, a monitoring method, and a monitoring program. [Background technology]

[0002] In Japan, the standard of living has improved, the sanitary environment has improved, and the level of medical care has improved, all of which have come with the rapid economic growth since the war. This, combined with the declining birth rate, has resulted in an aging society with a high aging rate. In such an aging society, it is expected that the number of people who require nursing care due to illness, injury, aging, etc. will increase.

[0003] In facilities such as hospitals and elderly care facilities, people who need care may fall while walking or fall off their beds and get injured. Therefore, in order to enable caregivers, nurses, and other staff to rush to the scene immediately depending on the condition of the person who needs care, development of a system that can detect the condition of the person who needs care from captured images is underway.

[0004] The following prior art is disclosed in Patent Document 1 below. When it is determined based on an image from an imaging device installed in the bed that the user is in a first state, such as when the user is inside the bed, the possibility of the user falling is predicted based on a first parameter, such as the user's posture. When it is determined based on an image from the imaging device that the user is in a second state, such as when the user is outside the bed, the possibility of the user falling is predicted based on a second parameter, such as the state of the bed's peripheral equipment. This makes it possible to notify the user of the risk of falling, etc. at an appropriate time. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-61829 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned prior art has a problem that it is cumbersome to predetermine and set parameters related to the risk of falling for each state of the care recipient, etc. Also, it has a problem that it cannot detect subtle changes in the behavior of the care recipient, etc.

[0007] The present invention has been made to solve such problems, and aims to provide a monitoring system, a monitoring method, and a monitoring program that can easily reduce the risk of a care-requiring person or the like falling and detect subtle changes in the behavior of the care-requiring person or the like. [Means for solving the problem]

[0008] The above-mentioned object of the present invention is achieved by the following means.

[0009] (1) A monitoring system having an acquisition unit that acquires an image of a specified area, an object state detection unit that detects a state of an object used by a subject in the acquired image, a calculation unit that calculates a normal state of the object based on a past state of the object, a change detection unit that detects a change in the detected state of the object from the calculated normal state, and a notification unit that notifies an alarm when the change is detected.

[0010] (2) A monitoring system as described in (1) above, comprising a subject state detection unit that detects the subject's lying down state based on the image, and the object state detection unit detects the state of the object when the subject's lying down state is detected.

[0011] (3) The monitoring system described in (1) above, wherein the state of the object is at least one of the position of the object and the angle of the object.

[0012] (4) The monitoring system described in (1) above, wherein the notification unit notifies the normal state of the object as the state to which the object should be returned when the change is detected.

[0013] (5) The monitoring system described in (1) above, wherein the object is a chair, a wheelchair, slippers, a chair-type portable toilet, or a movable handrail.

[0014] (6) The monitoring system described in (1) above, wherein the calculation unit calculates the normal state from the state of the object detected when the subject's lying down state was detected during a predetermined period in the past.

[0015] (7) A monitoring system as described in (1) above, comprising an event detection unit that detects a specified event in which the subject uses the object based on the image, and the calculation unit calculates the normal state from the state of the object detected immediately before the specified event was detected during a specified period in the past.

[0016] (8) The monitoring system described in (4) above, wherein the state of the object is the position of the object, and the notification unit notifies the normal position of the object by marking it on the image when the change is detected.

[0017] (9) A monitoring method executed by a monitoring system, comprising the steps of: (a) acquiring an image of a predetermined area; (b) detecting a state of an object used by a subject in the acquired image; (c) calculating a normal state of the object based on a past state of the object; (d) detecting a change in the detected state of the object relative to the calculated normal state; and (e) notifying an alarm when the change is detected.

[0018] (10) A monitoring method as described in (9) above, comprising a step (f) of detecting the subject's lying down state based on the image, and in the step (b), detecting the state of the object when the subject's lying down state is detected.

[0019] (11) The monitoring method described in (9) above, wherein the state of the object is at least one of the position of the object and the angle of the object.

[0020] (12) The monitoring method described in (9) above, wherein in step (e), when the change is detected, the normal state of the object is notified as the state to which the object should be returned.

[0021] (13) The monitoring method described in (9) above, wherein the object is a chair, a wheelchair, slippers, a chair-type portable toilet, or a movable handrail.

[0022] (14) The monitoring method described in (9) above, in which in the step (c), the normal state is calculated from the state of the object detected when the subject's lying down state is detected during a predetermined period in the past.

[0023] (15) A monitoring method as described in (9) above, comprising a step (g) of detecting a specified event in which the subject uses the object based on the image, and in the step (c), calculating the normal state from the state of the object detected immediately before the specified event was detected during a specified period in the past.

[0024] (16) The monitoring method described in (12) above, wherein the state of the object is the position of the object, and in step (e), when the change is detected, the normal position of the object is marked on the image to notify the change.

[0025] (17) A monitoring program for causing a computer to execute the monitoring method according to any one of (9) to (16) above. Effect of the Invention

[0026] In an image of a specific area, the state of an object used by a subject is detected, and the normal state of the object is calculated based on the past state of the object. Then, any change in the state of the object from the normal state is detected, and an alarm is issued when a change is detected. This makes it possible to easily reduce the risk of a care recipient falling, and to detect subtle changes in the behavior of the care recipient. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a system. [Diagram 2] FIG. 2 is a block diagram showing a configuration of the imaging device. [Diagram 3] FIG. 2 is a block diagram showing the configuration of a server. [Figure 4] FIG. 2 is a functional block diagram of a control unit. [Diagram 5] FIG. 2 shows a person rectangle and an object rectangle detected in an image. [Figure 6A] FIG. 13 is a diagram showing an example of an object rectangle. [Figure 6B] FIG. 13 is a diagram showing an example of an object rectangle. [Figure 6C] FIG. 13 is a diagram showing an example of an object rectangle. [Figure 6D] FIG. 13 is a diagram showing an example of an object rectangle. [Figure 7] FIG. 2 is a schematic diagram showing joint points. [Figure 8] FIG. 13 is a schematic diagram showing joint points detected based on a human rectangle; [Figure 9] FIG. 11 is an explanatory diagram for explaining an angle of an object rectangle. [Figure 10] FIG. 2 is an explanatory diagram showing an example of a state of an object to be detected. [Figure 11] 11A and 11B are explanatory diagrams showing other examples of the state of an object to be detected. [Figure 12A] FIG. 11 is an explanatory diagram showing still another example of the state of an object to be detected. [Figure 12B] FIG. 11 is an explanatory diagram showing still another example of the state of an object to be detected. [Figure 13] FIG. 2 is a block diagram showing a configuration of a mobile terminal. [Figure 14] FIG. 13 is a diagram showing an example of a display screen of an alarm displayed on a mobile terminal. [Figure 15] FIG. 13 is a diagram showing another example of a display screen of an alarm displayed on the mobile terminal. [Figure 16] 4 is a flowchart showing the operation of the system. [Figure 17] 4 is a flowchart showing the operation of the system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, a monitoring system, a monitoring method, and a monitoring program according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation, and may differ from the actual ratios.

[0029] FIG. 1 is a diagram showing a schematic configuration of a system 10. As shown in FIG.

[0030] The system 10 includes a photographing device 100, a server 200, a communication network 300, and a mobile terminal 400. The photographing device 100 is connected to the server 200 via the communication network 300 so that they can communicate with each other. The mobile terminal 400 can be connected to the communication network 300 via an access point 310. The server 200 constitutes a monitoring system. Note that the photographing device 100 may be configured to execute some or all of the functions of the server 200. In this case, the photographing device 100 alone or together with the server 200 may constitute a monitoring system.

[0031] (imaging device 100) Fig. 2 is a block diagram showing the configuration of the photographing device 100. As shown in the example of Fig. 2, the photographing device 100 includes a control unit 110, a communication unit 120, and a camera 130, which are connected to each other via a bus.

[0032] The control unit 110 is composed of a CPU (Central Processing Unit) and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and controls each part of the image capturing device 100 and performs calculation processing according to a program. The control unit 110 transmits an image 600 (see FIG. 5) obtained by the camera 130 capturing an image of a predetermined area to the server 200 or the like via the communication unit 120. Hereinafter, the image 600 captured by the camera 130 will also be simply referred to as the "image 600." The predetermined area is, for example, a three-dimensional area including the entire floor surface of the subject's 500 room.

[0033] The communication unit 120 is an interface circuit (eg, a LAN card) for communicating with the mobile terminal 400 and the like via the communication network 300.

[0034] The camera 130 is, for example, a wide-angle camera. The camera 130 is installed at a position overlooking a predetermined area by installing the image capturing device 100 on the ceiling or the like of a room of a subject 500 such as a person requiring care, and captures the predetermined area. The subject 500 is, for example, a person who requires care such as nursing care or nursing care from a staff member or the like. The camera 130 may be a standard camera with a narrower angle of view than a wide-angle camera. The image 600 may include the subject 500 (i.e., a person) and an object 700 such as a bed 710 and a chair 720 as images. The image 600 includes still images and videos. The camera 130 is a near-infrared camera, and may capture a predetermined area by irradiating near-infrared rays toward the capture area using an LED (Light Emitting Device) and receiving the near-infrared reflected light reflected by an object in the capture area using a CMOS (Complememtary Metal Oxide Semiconductor) sensor. The image 600 may be a monochrome image in which each pixel has a reflectance of near-infrared rays. The camera 130 may be a visible light camera instead of a near-infrared camera, or a combination of the two may be used.

[0035] (Server 200) 3 is a block diagram showing the configuration of server 200. Server 200 includes a control unit 210, a communication unit 220, and a storage unit 230. The components are connected to each other via a bus.

[0036] The basic configurations of the control unit 210 and the communication unit 220 are similar to those of the control unit 110 and the communication unit 120, which are the corresponding components of the imaging device 100, and therefore will not be described repeatedly. The storage unit 230 is composed of a RAM, a ROM, a HDD (Hard Disc Drive), etc.

[0037] 4 is a functional block diagram of the control unit 210. The control unit 210 functions as an acquisition unit 211, an object detection unit 212, a subject state detection unit 213, an object state detection unit 214, a calculation unit 215, a change detection unit 216, an event determination unit 217, and a notification unit 218. The subject state detection unit 213 and the event determination unit 217 configure an event detection unit.

[0038] The acquiring unit 211 acquires an image 600 in which a predetermined area is captured by the image capturing device 100. Specifically, the acquiring unit 211 acquires the image 600 by receiving the image 600 from the image capturing device 100 via the communication unit 220. If the image 600 received from the image capturing device 100 is stored in the storage unit 230, the acquiring unit 211 may acquire the image 600 by reading it from the storage unit 230. The image 600 includes an image 600 including a subject 500 and an object 700. The object 700 includes, for example, a wheelchair 730, a walker, slippers 740, a chair 720, a chair-type portable toilet, a movable handrail 750, and a bed 710 (see FIGS. 6A to 6D).

[0039] The object detection unit 212 detects the target person 500 and the object 700 in the image 600. Specifically, the object detection unit 212 detects the target person 500 and the object 700, for example, as follows. The object detection unit 212 detects rectangular areas including the target person 500 and the object 700 from the image 600, and identifies (estimates) an attribute class for each rectangular area. The attribute class is a classification of the person and each object 700. The attribute classes are, for example, "person", "chair", "wheelchair", "slippers", "chair-type portable toilet", and "movable handrail". Hereinafter, a rectangular area in which the attribute class of "person" is identified is referred to as a person rectangle 610. A rectangular area in which the attribute class of an object 700 other than "person" is identified is referred to as an object rectangle 630. The object detection unit 212 detects the person rectangle 610 as the target person 500. A trained model of a neural network may be used to detect rectangular regions and identify attribute classes in the image 600. An example of the trained model is the Region Proposal Network (RPN), which is a neural network model.

[0040] Fig. 5 is a diagram showing a person rectangle 610 and an object rectangle 630 detected in an image 600. In the example of Fig. 5, an object rectangle 630 with an attribute class of "chair" is shown.

[0041] 6A to 6D are diagrams showing examples of an object rectangle 630. The object rectangle 630 may include a center line of the object rectangle 630. The center line of the object rectangle 630 may be used to calculate the angle of the object rectangle 630, which will be described later. FIG. 6A shows an example of an object rectangle 630 of a chair 720. FIG. 6B shows an example of an object rectangle 630 of a wheelchair 730. FIG. 6C shows an example of an object rectangle 630 of a slipper 740. FIG. 6D shows an example of an object rectangle 630 of a movable handrail 750.

[0042] The subject state detection unit 213 detects the state (behavior) of the subject 500 based on the image 600. For example, the subject state detection unit 213 detects the state of the subject 500 based on a human rectangle 610 detected from the image 600. The subject state detection unit 213 may detect the state of the subject based on the human rectangle 610 and the object rectangle 630 detected from the image 600.

[0043] For ease of explanation, the following description will be given taking as an example a case where the state of the subject 500 is a state caused by the subject 500's sole action.

[0044] Specifically, the subject state detection unit 213 detects the state of the subject 500, for example, as follows: The subject state detection unit 213 estimates joint points 620 of the subject 500 based on the human rectangle 610 (see FIGS. 7 and 8, etc.).

[0045] FIG. 7 is a schematic diagram showing an articulation point 620.

[0046] In the example of FIG. 7, the joint points 620 include a head joint point 620a, a shoulder joint point 620b, a waist joint point 620c, and a joint point 620d constituting the trunk.

[0047] FIG. 8 is a schematic diagram showing articulation points 620 detected based on a human rectangle 610.

[0048] 8, the articulation points 620 of the subject 500 in a lying position are shown. That is, the articulation points 620 of the subject 500 in a lying position on the bed 710 are shown.

[0049] The subject state detection unit 213 may estimate the joint points 620 using a trained model of a neural network that has been trained to detect the joint points 620 from the human rectangle 610. As the trained model, a known model such as Deep Pose may be used. Details of DeepPose are described in a known document (Alexander Toshev, et al. "DeepPose: Human Pose Estimation via Deep Neural Networks", in CVPR, 2014).

[0050] The subject state detection unit 213 detects the state (behavior) of the subject 500 based on the joint point 620. For example, when the subject state detection unit 213 detects the subject 500 lying on the bed 710 from the joint point 620, the subject state detection unit 213 detects the state as "lying". For example, when the object rectangle 630 of the wheelchair 730 and the person rectangle 610 overlap, the subject state detection unit 213 detects the state of "transfer". For example, when the bed 710 and the person rectangle 610 change from an overlapping state to a non-overlapping state, the subject state detection unit 213 detects the state of "getting out of bed". The position and range of the bed 710 may be set in advance in the coordinate system of the coordinates of the joint point 620. The bed 710 may be detected as the object rectangle 630. For example, when the subject 500 is detected to be in a lying position within a short time from a state in which the subject 500 is detected to be in an upright position from the joint points 620, the subject state detection unit 213 detects the state of "falling". The state of the subject 500 may be detected using a trained model of a neural network. The trained model is a model trained to estimate the state of the subject 500 based on a plurality of joint points 620 estimated from a plurality of images 600 adjacent in time series.

[0051] The object state detection unit 214 detects the state of the object 700 used by the subject 500 in the image 600. For example, the object state detection unit 214 detects the state of the object 700 based on the object rectangle 630 detected from the image 600. It is preferable that the object state detection unit 214 detects the state of the object 700 that is not fixed and comes into contact with the body of the subject 500 when in use. For example, the object state detection unit 214 detects at least one of the states of the chair 720, the wheelchair 730, the slippers 340, the chair-type portable toilet, and the movable handrail 750. The state of the object 700 may be at least one of the position of the object 700 and the angle of the object 700.

[0052] Specifically, the object state detection unit 214 detects the state of the object 700, for example, as follows. The object state detection unit 214 detects the center coordinates of the object rectangle 630 as the state of the object (position of the object). The object state detection unit 214 detects the angle of the object rectangle 630 with respect to a predetermined direction (hereinafter also referred to as the "angle of the object rectangle 630") as the state of the object (angle of the object). The predetermined direction may be the direction of any coordinate axis in the coordinate system of the image 600. The predetermined direction may be a direction parallel to the side of the bed 710 placed in place.

[0053] Fig. 9 is an explanatory diagram for explaining the angle of the object rectangle 630. Note that Fig. 9 also shows an average value 631 of the object rectangle 630 of the chair 720 for a predetermined period in the past, which is a normal state of the object 700, as described later.

[0054] As described above, the object rectangle 630 may include a center line (vector) having directional information. For example, in the case of the object rectangle 630 of the chair 720, the directional information may be information about the direction from the back to the front legs. This allows a distinction to be made between the angle of the object rectangle 630 of the chair 720 being 0 degrees and the angle of 180 degrees. The angle of the object rectangle 630 is the angle between the center line of the object rectangle 630 and the direction parallel to the side of the bed 710.

[0055] The calculation unit 215 calculates the normal state of the object 700 based on the past state of the object 700. The calculation unit 215 can calculate the normal state from the state of the object 700 detected when the subject 500 is lying down, which is detected in a past predetermined period. For example, the normal state is the average value or median value of the position or angle of the object 700 detected when the subject 500 is lying down, which is detected in a past predetermined period. The reason why the normal state of the object 700 is calculated based on the state of the object 700 when the subject 500 is lying down is because the state of the object 700 when the subject 500 is lying down is most likely to be the normal state. In addition, when the subject 500 transitions from a lying down state to a getting up state, the state of the object 700 that is likely to be touched first is relatively highly related to the risk of falling or falling. The past predetermined period is set to an appropriate period from the viewpoints of accuracy, shortening of calculation time, and the like. The predetermined period may be any period between one week and one year, for example.

[0056] On the other hand, the calculation unit 215 may calculate the normal state of the object 700 from the state of the object 700 detected immediately before a predetermined event is detected by the event determination unit 217 during a past predetermined period. The predetermined event may be an event in which the subject 500 uses the object 700. The predetermined event may be, for example, "transfer" and "getting out of bed." In the "transfer" event, the wheelchair 730, the chair 720, and a chair-type portable toilet, etc. may be used. In the "getting out of bed" event, the movable handrail 750, the slippers 740, etc. may be used. The reason why the normal state of the object 700 is calculated based on the state of the object 700 detected immediately before the detection of the predetermined event is that the state of the object 700 immediately before being used in the predetermined event is likely to be in a normal state. In addition, the state of the object 700 that the subject 500 is likely to touch first during the predetermined event is relatively highly related to the risk of falling, tripping, etc.

[0057] The calculation section 215 may store information on the calculated normal state in the storage section 230.

[0058] The change detection unit 216 detects whether or not there is a change in the state of the object 700 detected by the object state detection unit 214 with respect to the normal state. Specifically, the changes detected by the change detection unit 216 are, for example, the following: (1) When the normal state is the average value of the object rectangle 630 for a predetermined past period (average values ​​of the coordinates of the vertices of the object rectangle 630), there is no range in which the average value of the object rectangle 630 in the normal state overlaps with the current object rectangle 630. (2) When the normal state is the average value of the center coordinates of the object rectangle 630 for a predetermined past period, the distance between the average value of the center coordinates of the object rectangle 630 in the normal state and the center coordinates of the current object rectangle 630 is a predetermined value or more. (3) When the normal state is the average value of the angle of the object rectangle 630 for a predetermined past period, the difference between the average value of the angle of the object rectangle 630 in the normal state and the angle of the current object rectangle 630 is a predetermined value or more.

[0059] 10 is an explanatory diagram showing an example of the state of a detected object 700. The dashed rectangle is the average value 631 of the object rectangle 630 over a past predetermined period, which is a normal state. The solid rectangle is the current object rectangle 630.

[0060] 10 shows a state in which, when the normal state is the average value of the object rectangle 630 for a predetermined period of time in the past, there is no overlapping range between the average value 631 of the object rectangle 630 in the normal state and the current object rectangle 630. The change detection unit 216 detects this state as a change in the state of the object 700 from the normal state.

[0061] 11 is an explanatory diagram showing another example of the state of the detected object 700. The dashed rectangle is the average value 631 of the object rectangle 630 for a predetermined period of the past. The point inside the dashed rectangle is the average value of the center coordinates of the object rectangle 630 for a predetermined period of the past, which is a normal state. The solid rectangle is the current object rectangle 630. The point inside the solid rectangle is the center coordinates of the current object rectangle 630.

[0062] Fig. 11 shows an example in which the normal state is the average value of the center coordinates of the object rectangle 630 for a predetermined period in the past. Fig. 11 shows a state in which the distance between the average value of the center coordinates of the object rectangle 630 in the normal state and the center coordinates of the current object rectangle 630 is equal to or greater than a predetermined value. The length of the double-headed arrow in Fig. 11 indicates the distance between the average value of the center coordinates of the object rectangle 630 in the normal state and the center coordinates of the current object rectangle 630. The change detection unit 216 detects this state as a change in the state of the object 700 from the normal state. The predetermined value can be appropriately set by experiment or the like in terms of the detection accuracy of the change.

[0063] 12A and 12B are explanatory diagrams showing still another example of the state of the detected object 700. The dashed rectangle in FIG. 12A is the average value 631 of the object rectangle 630 in the past predetermined period. The point in the dashed rectangle is the average value of the center coordinates of the object rectangle 630 in the past predetermined period. The angle α is the average value of the angle of the object rectangle 630 in the past predetermined period, which is a normal state. The angle is the angle between the center line of the object rectangle 630 and the direction parallel to the side of the bed 710. The solid rectangle in FIG. 12B is the current object rectangle 630. The point in the solid rectangle is the center coordinates of the current object rectangle 630. The angle β is the angle of the current object rectangle 630.

[0064] 12A and 12B show an example in which the normal state is angle α, which is the average value of the angles of the object rectangle 630 over a predetermined period of time in the past. FIG. 12A and 12B show a case in which the difference between the angle α, which is the normal state, and the angle β of the current object rectangle 630 is equal to or greater than a predetermined value. The change detection unit 216 detects this state as a change in the state of the object 700 from the normal state. The predetermined value can be appropriately set through experiments or the like in terms of the detection accuracy of the change.

[0065] The event determination unit 217 determines whether or not the state of the subject 500 detected by the subject state detection unit 213 corresponds to a predetermined event. As described above, the predetermined event is an event in which the subject 500 uses the object 700. Examples of the predetermined event are "transfer" and "getting out of bed."

[0066] The notification unit 218 notifies the mobile terminal 400 of an alarm when the change detection unit 216 detects a change in the state of the object 700 from the normal state. The notification unit 218 may notify a terminal other than the mobile terminal 400 of the alarm. The alarm includes a simple warning and a warning including information for notifying that the position of the object 700 is shifted. Furthermore, the alarm includes a warning including information for notifying that the position of the object 700 is shifted by comparing and displaying the normal state and the current state of the object 700. The alarm may include guidance information for comparing and displaying the normal state and the current state of the object 700 to guide the current state to return to the normal state. That is, when a change from the normal state is detected, the notification unit 218 may notify the normal state of the object 700 as the state to which the object 700 should be returned on the image 600. Furthermore, when the state of the object 700 is detected as the position of the object 700, the normal position of the object may be marked on the image 600 to be notified.

[0067] The notification unit 218 can notify the mobile terminal 400 of the state of the subject 500 detected by the subject state detection unit 213.

[0068] (Mobile terminal 400) 13 is a block diagram showing the configuration of mobile terminal 400. Mobile terminal 400 includes control unit 410, wireless communication unit 420, display unit 430, input unit 440, and audio input / output unit 450. The components are connected to each other via a bus. Mobile terminal 400 may be configured by a communication terminal device such as a tablet computer, a smartphone, or a mobile phone, for example.

[0069] The control unit 410 includes basic components such as a CPU, RAM, and ROM, similar to the configuration of the control unit 110 of the photographing device 100. The control unit 410 controls each unit of the mobile terminal 400 and performs calculation processing according to a program.

[0070] The wireless communication unit 420 has a function of performing wireless communication according to standards such as Wi-Fi and Bluetooth (registered trademark), and wirelessly communicates with each device directly or via the access point 310.

[0071] The wireless communication unit 420 receives an alarm from the server 200. As described above, the alarm is transmitted from the server 200 to the mobile terminal 400 when a change in the state of the object 700 from the normal state is detected.

[0072] The display unit 430 and the input unit 440 are touch panels, and a touch sensor serving as the input unit 440 is provided on the display surface of the display unit 430 which is made up of a liquid crystal or the like.

[0073] The display unit 430 displays the alarm received from the server 200 .

[0074] FIG. 14 is a diagram showing an example of a display screen of an alarm displayed on mobile terminal 400. As shown in FIG.

[0075] 14, an alarm including a warning message such as "The chair is out of position" is displayed on the image 600. In addition, in the image 600, the chair 720 in which a change from the normal state has been detected is highlighted by being surrounded by a thick frame. The alarm display screen in FIG. 14 is displayed, for example, when an alarm including information notifying that the object 700 is out of position is received.

[0076] FIG. 15 is a diagram showing another example of the display screen of the alarm displayed on the mobile terminal 400. In FIG.

[0077] On the alarm display screen in Fig. 15, chair 720, in which a change from the normal state has been detected on image 600, is highlighted by being surrounded by a thick frame. In addition, the normal state (position) of chair 720 is displayed as a dashed rectangle. That is, the normal state (position) of chair 720 is displayed by being marked with a dashed rectangle. This allows a comparison between the normal state and the current state of object 700. Furthermore, guidance information such as "Please return to the position of the dashed line" is displayed together with an arrow.

[0078] The voice input / output unit 450 is, for example, a speaker and a microphone, and enables voice communication between staff members with other mobile terminals 400 via the wireless communication unit 420 .

[0079] For example, the audio input / output unit 450 may issue a simple warning alarm received from the server 200 by an alarm sound. For example, the audio input / output unit 450 may issue an alarm of warning information for notifying that the position of the object 700 is shifted received from the server 200 by outputting the words "The position of the chair is shifted" in a voice.

[0080] The operation of the system 10 will now be described.

[0081] 16 and 17 are flowcharts showing the operation of the system 10. FIG. 17 is a subroutine flowchart of step S103 in the flowchart of FIG. 16. These flowcharts may be executed by the control unit 210 of the server 200 in accordance with a program. When some or all of the functions shown in FIG. 4 are executed by the photographing device 100, some or all of this flowchart may be executed by the control unit 110 of the photographing device 100 in accordance with a program. The order of each step does not have to be the order shown in FIGS. 16 and 17, and steps may be executed in parallel.

[0082] The control unit 210 acquires the image 600 by receiving it from the photographing device 100 (S101).

[0083] The control unit 210 detects the target person 500 and the object 700 in the image 600 (S102).

[0084] The control unit 210 calculates the normal state of the detected object 700 (S103). Details of step S103 will be described below with reference to the subroutine flowchart of Fig. 17. Note that the process of step S103 may be performed in advance.

[0085] The control unit 210 acquires a previous image 600 (taken in the past) relative to the image 600 acquired in step S101 (S201). Thereafter, steps S201 to S206 are repeatedly executed for all images taken within a predetermined period in the past.

[0086] The control unit 210 detects the target person 500 and the object 700 in the image 600 (S202).

[0087] The control unit 210 detects the state of the detected target person 500 (S203).

[0088] The control unit 210 determines whether the state of the subject 500 detected in step S203 is a state of lying down (S204). When the control unit 210 determines that the state of the subject 500 is not a state of lying down (S204: NO), the control unit 210 returns to step S201 and continues the process.

[0089] When the control unit 210 determines that the state of the subject 500 is lying in bed (S204: YES), the control unit 210 detects the state of the object 700 detected in step S202 (S205). Furthermore, the control unit 210 stores the detected state of the object 700 in the storage unit 230.

[0090] The control unit 210 determines whether all of the images 600 taken during the past predetermined period have been acquired in step S201 (S206). If the control unit 210 determines that all of the images 600 taken during the past predetermined period have not been acquired in step S201 (S206: NO), the control unit 210 returns to step S201 and continues the process.

[0091] When the control unit 210 determines that all of the images 600 captured during the past predetermined period have been acquired in step S201 (S206: YES), the control unit 210 executes step S207. The control unit 210 calculates the normal state of the object 700 based on the state of the object 700 stored in the storage unit 230 in step S205 (S207). For example, the control unit 210 calculates the average value or median value of the position or angle of the object 700, which is the state of the object 700, as the normal state.

[0092] Returning to FIG. 16, the description will continue.

[0093] The control unit 210 detects the state of the object 700 detected in step S102 (S104).

[0094] The control unit 210 detects a change in the state of the object 700 from the normal state (S105).

[0095] When the control unit 210 determines that a change from the normal state of the object 700 has not been detected (S106: NO), the control unit 210 returns to step S101 and continues the process.

[0096] When the control unit 210 determines that a change from the normal state of the object 700 has been detected (S106: YES), it notifies an alarm to the mobile terminal 400 or the like (S107).

[0097] The embodiment provides the following advantages.

[0098] In an image in which a specified area is captured, the state of an object used by a subject is detected, and the normal state of the object is calculated based on the past state of the object. Then, a change in the state of the object relative to the normal state is detected, and an alarm is issued when a change is detected. This makes it possible to easily reduce the risk of a person requiring care falling, and to detect subtle changes in the behavior of the person requiring care. By detecting subtle changes in the behavior of the person requiring care, for example, signs of dementia can be discovered early.

[0099] Furthermore, based on the image, the state of the subject lying down is detected, and the state of the object when the state of the subject lying down is detected is detected. This improves the accuracy of calculating the normal state of the object. Also, it is possible to reduce the risk of a care-requiring person, etc., falling, which is relatively high when transitioning from a lying down state to the next state.

[0100] In addition, the state of the object is determined to be at least one of the position of the object and the angle of the object, which can further reduce the risk of the care-requiring person or the like falling and can improve the accuracy of detecting subtle changes in the behavior of the care-requiring person or the like.

[0101] Furthermore, when a change in the state of an object from the normal state is detected, the normal state of the object is notified as the state to which the object should be returned, making it possible to easily guide the object back to its normal position.

[0102] In addition, the detected objects are chairs, wheelchairs, slippers, portable chair-type toilets, or movable handrails, making it easier to reduce the risk of falls for people in need of care.

[0103] In addition, the normal state is calculated from the state of the object detected during the past predetermined period when the subject was lying down, thereby improving the accuracy of calculating the normal state of the object.

[0104] In addition, a predetermined event in which the subject uses an object is detected based on the image. Then, a normal state is calculated from the state of the object detected immediately before the predetermined event is detected during a past predetermined period. This improves the accuracy of calculating the normal state. In addition, the risk of a fall of a person requiring care, which is relatively high when a predetermined event is detected, can be reduced.

[0105] In addition, when a change in the state of an object relative to its normal state is detected, the normal state of the object is marked on the image to notify the user, making it easier to guide the object back to its normal position.

[0106] The system configuration described above is a main configuration for explaining the features of the above embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims. In addition, the configuration of a general monitoring system is not excluded.

[0107] In addition, the means and methods for performing various processes in the above-mentioned system can be realized by either a dedicated hardware circuit or a programmed computer. The above-mentioned program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The above-mentioned program may be provided as a standalone application software, or may be incorporated as a function into the software of a server or other device. [Explanation of symbols]

[0108] 10 systems, 100 imaging device, 110 control section, 120 Communications Department, 130 cameras, 200 servers, 210 control section, 211 Acquisition Department; 212 object detection unit, 213 subject state detection unit, 214 object state detection unit, 215 Calculation unit, 216 Conversion detection unit, 217 Event determination unit, 218 Notification Department; 220 Communications Department, 230 storage section, 300 communication networks, 400 mobile devices, 500 subjects (people), 600 images, 610 people rectangle, 620 articulation points, 630 object rectangle, 631 Average value of object rectangles for a given period of time (normal state of the object), 700 objects, 710 beds, 720 chairs, 730 wheelchairs, 740 slippers, 750 Movable handrails.

Claims

1. An acquisition unit that acquires an image of a predetermined area; An object state detection unit that detects a state of an object used by a subject in the acquired image; A calculation unit that calculates a normal state of the object based on a past state of the object; a change detection unit that detects a change in the detected state of the object relative to the calculated normal state; a notification unit that issues an alarm when the change is detected; A monitoring system having the above.

2. A subject state detection unit that detects the subject's lying state based on the image, The monitoring system according to claim 1 , wherein the object state detection unit detects the state of the object when the subject's lying down state is detected.

3. The monitoring system according to claim 1 , wherein the state of the object is at least one of a position of the object and an angle of the object.

4. The monitoring system according to claim 1 , wherein the notification unit notifies the object of the normal state as a state to which the object should be returned when the change is detected.

5. The monitoring system according to claim 1 , wherein the object is a chair, a wheelchair, slippers, a chair-type portable toilet, or a movable handrail.

6. The monitoring system according to claim 1 , wherein the calculation unit calculates the normal state from a state of the object detected during a predetermined period in the past when the subject's lying down state was detected.

7. an event detection unit that detects a predetermined event in which the subject uses the object based on the image; The monitoring system according to claim 1 , wherein the calculation unit calculates the normal state from a state of the object detected immediately before the predetermined event was detected during a past predetermined period.

8. the state of the object is the position of the object; The monitoring system according to claim 4 , wherein the notification unit notifies the user of the normal position of the object by marking the normal position of the object on the image when the change is detected.

9. A monitoring method executed by a monitoring system, A step (a) of acquiring an image of a predetermined area; (b) detecting a state of an object used by a subject in the acquired image; (c) calculating a normal state of the object based on a past state of the object; (d) detecting a change in the sensed state of the object relative to the calculated normal state; (e) issuing an alarm when the change is detected; A monitoring method having the above structure.

10. (f) detecting a lying state of the subject based on the image; The monitoring method according to claim 9 , wherein in the step (b), a state of the object is detected when the subject's lying down state is detected.

11. The monitoring method according to claim 9 , wherein the state of the object is at least one of a position of the object and an angle of the object.

12. 10. The monitoring method according to claim 9, wherein in said step (e), when said change is detected, said normal state of said object is notified as a state to which said object should be returned.

13. The monitoring method according to claim 9 , wherein the object is a chair, a wheelchair, slippers, a chair-type portable toilet, or a movable handrail.

14. The monitoring method described in claim 9, wherein in step (c), the normal state is calculated from the state of the object detected when the subject's lying down state is detected during a predetermined period in the past.

15. (g) detecting a predetermined event in which the subject uses the object based on the images; The monitoring method according to claim 9 , wherein in the step (c), the normal state is calculated from a state of the object detected immediately before the specified event was detected during a specified past period.

16. the state of the object is the position of the object; The monitoring method according to claim 12 , wherein in said step (e), when said change is detected, said normal position of said object is notified by marking it on said image.

17. A monitoring program for causing a computer to execute the monitoring method according to any one of claims 9 to 16.

Images