Specific Action Detection Device

Abstract

To provide a specific operation detection device that determines patient's bed-leaving, with the inclusion of the presence of action that the patient sits up on a bed and sits on the bed.SOLUTION: A device has: a camera 1 photographing an entire monitoring area; a bed detection unit 3 that extracts a bed from the picked up image of the camera 1 to conduct area calculation of the bed from the extracted information; a person attitude detection unit 4 that obtains coordinate information of a specific part of a patient on the bed from the picked up image of the camera 1 to detect patient's attitude from the information; a relative position calculation unit 6 that calculates a relative position of the patient with the bed based on the information detected by the bed detection unit 3 and the person attitude detection unit 4; a relative position information holding unit 6 that holds time series relative position information calculated by the relative position calculation unit 5; and an action estimation unit 7 that estimates an action of the patient on the bed based on plural pieces of time series information that the relative position information holding unit 6 holds.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a specific action detection device that estimates a specific action of a person based on image information captured by a camera. [Background technology]

[0002] In recent years, there have been many accidents in hospitals and elderly care facilities where patients or residents fall while walking or moving. To prevent such accidents, there are systems that notify nurses or caregivers if they detect any movement immediately before getting out of bed. For example, in Patent Document 1, if it is detected that a patient has sat up in bed and puts his / her feet off the bed and sits down, a notification is sent to relevant persons. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-87040 A [Non-patent literature]

[0004] [Non-Patent Document 1] Madoka Inoue, Toshiyuki Yasui, Ryo Taguchi, and Taizo Umezaki: "Bed position detection for automation of patient monitoring functions", Journal of Electrical Engineering C, Vol. 138, No. 6, pp. 670-677 (2018) [Non-Patent Document 2] Z. Cao, T. Simon, SE Wei, & Y. Sheikh: “Realtime multi-person 2d pose estimation using part affinity fields”, In IEEE Conference on Computer Vision and Pattern Recognition, Vol.1, No.2 p.7 (2017) Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of the above-mentioned Patent Document 1 can reduce false positives by detecting a person moving away from the bed without sitting on it and determining that the person is not a patient. However, although it was possible to determine that the person was sitting on the bed, it did not determine whether this was a series of actions in which the person sat on the bed after getting up from the bed, which resulted in false positives.

[0006] In view of the above problems, the present invention aims to provide a specific motion detection device that can determine whether a patient has gotten out of bed, including whether or not the patient has sat up on the edge of the bed. [Means for solving the problem]

[0007] In order to solve the above problem, the specific behavior detection device according to the invention of claim 1 comprises a camera that captures an image of the entire monitoring area, a bed detection unit that extracts a bed from an image captured by the camera and calculates the area of ​​the bed from the extracted information, a person posture detection unit that obtains coordinate information of a specific part of a person on the bed from the image captured by the camera and detects the posture of the person from that information, a relative position calculation unit that obtains the position of the center of gravity of the bed based on the information detected by the bed detection unit and the person posture detection unit and calculates the relative position between the bed and the person, a relative position information storage unit that stores the relative position information calculated by the relative position calculation unit in chronological order, and Using neural networks and a behavior estimation unit for estimating the behavior of a person on the bed. The relative position calculation unit calculates a rectangular reference rectangle based on the diamond-shaped bed information detected by the bed detection unit from the captured image of the camera, and calculates the relative position of the patient from the reference rectangle, the center of gravity of the bed, and the patient's posture detected by the human posture detection unit. It is characterized by: With this configuration, the behavior of a person can be estimated by observing the posture of the person on the bed and the change in the relative position between the bed and the person. In other words, if it is detected that the monitored patient gets up on the bed and then sits on the edge of the bed, the person moving away from the bed is determined to be the patient getting out of bed. This reduces erroneous detections in which the bed-leaving action of a person who is not the monitored person, such as a nurse, is determined to be the patient getting out of bed.

[0008] The invention of claim 2 is characterized in that, in the configuration described in claim 1, the human posture detection unit obtains coordinate information of at least three points of the neck, left pelvis, and right pelvis out of the head, neck, left hand, right hand, left pelvis, and right pelvis of the person on the bed, and estimates the posture based on the coordinate information. According to this configuration, by estimating a person's posture based on skeletal information, it is possible to detect with high accuracy a state in which the person sits up in bed or sits on the edge of the bed, and to estimate the person's posture. Effect of the Invention

[0009] According to the present invention, the behavior of a person can be estimated by observing the posture of the person on the bed and the change in the relative position between the bed and the person. That is, when a monitored patient gets up on the bed and then sits down on the bed, the person moving away from the bed is judged to be the patient's leaving the bed. This reduces the erroneous judgment that the leaving action of a person who is not the monitored person, such as a nurse, is the patient's leaving the bed. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a specific action detection device according to the present invention; [Diagram 2] FIG. 13 is a diagram showing a detected bed and a reference rectangle. [Diagram 3] FIG. 13 is an explanatory diagram outlining the processing, in which (a) shows a captured image of a patient, and (b) shows the positional relationship between the patient and the bed calculated from the captured image. [Figure 4] FIG. 1 is an explanatory diagram of a neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a block diagram showing an example of a specific action detection device according to the present invention, in which 1 is a camera that captures an image of the entire monitoring area, 2 is an image storage unit that temporarily stores an image captured by camera 1, 3 is a bed detection unit that detects a bed from the image captured by camera 1, 4 is a person posture detection unit that detects a posture of a person from the image captured by camera 1, 5 is a relative position calculation unit that calculates the relative position of the person with respect to the detected bed, 6 is a relative position information storage unit that stores the calculated relative position information, and 7 is a behavior estimation unit that estimates the behavior of the person.

[0012] The person referred to here is a person to be monitored, such as a patient or a person receiving care, and will be referred to hereinafter as a "patient." The image storage unit 2, bed detection unit 3, person posture detection unit 4, relative position calculation unit 5, relative position information storage unit 6, and behavior estimation unit 7 can be integrated into a CPU or DSP with an operating program installed, or can be executed by a personal computer.

[0013] The camera 1 is installed on a wall, ceiling, or the like so as to capture an image of the entire room, such as a hospital room, in order to capture an image of a patient, and generates and outputs an image frame, for example, every 0.03 seconds.

[0014] The image storage unit 2 is composed of, for example, a RAM, and temporarily stores the image (image frame) captured by the camera 1 until the image is processed by the bed detection unit 3 or the person posture detection unit 4 in the subsequent stage.

[0015] The bed detection unit 3 first extracts the bed from the image captured by the camera 1, and calculates the four coordinates of the four corners of the extracted bed. As a method for extracting the bed, for example, the machine learning-based bed position estimation method described in Non-Patent Document 1 can be used.

[0016] The human posture detection unit 4 estimates the posture of the patient from the image captured by the camera 1. To estimate the sitting position on the edge of the bed, the posture is estimated from the coordinate information of at least three points: the neck, the left pelvis, and the right pelvis. For this estimation, the method described in Non-Patent Document 2 can be used. Here, the posture is estimated based on the coordinate information of three points, namely the neck, left pelvis, and right pelvis. However, it may also be estimated based on four skeletal points by adding the coordinates of the head, or the posture may be estimated by further adding the coordinate information of the left and right hands, etc.

[0017] The relative position calculation unit 5 converts the coordinate information obtained from the image into relative values. This is performed because the patient's behavior cannot be determined with high accuracy even if it is estimated directly from the bed coordinates and the patient's posture. The procedure is as follows: first, calculate the center of gravity of the bed from the calculated four-point coordinates of the bed. Next, calculate a reference rectangle from the bed coordinates. The reference rectangle is a diamond-shaped bed because the image shows 3D information in 2D, but in order to handle it as 2D information, it is recalculated to be a rectangle that matches the actual bed shape.

[0018] In this way, input data can be reduced by converting relative coordinates, and the positional relationship between the bed and the patient can be properly expressed, leading to improved detection accuracy, even if the bed position on the screen changes due to a change in the position of the camera 1. Also, by converting the coordinate information into relative values ​​instead of directly inputting it, it becomes possible to detect the patient's behavior with higher accuracy than in the past, when the patient's behavior was directly estimated from the bed coordinates and the patient's posture.

[0019] After calculating the center of gravity of the bed and the reference rectangle, the distance from the center of gravity of the bed to the reference rectangle is then calculated for each of the top, bottom, left, and right directions. Figure 2 shows the coordinate relationship between the bed and the reference rectangle thus calculated. In Figure 2, D1 is the bed, D2 is the reference rectangle, Gb is the center of gravity of the bed, rtx is the x-coordinate of the upper right, rbx is the x-coordinate of the lower right, and rx is the x-coordinate of the reference rectangle, and rx is calculated using the following formula. rx=(rtx+rbx) / 2 Furthermore, the four corners of the bed are expressed by four (x, y) coordinates (a total of eight variables), whereas the reference rectangle is two-dimensional and can be expressed by a total of four variables: top, bottom, left, and right.

[0020] In this way, the relative position of the patient is calculated from the position of the center of gravity Gb of the bed D1 and the reference rectangle D2 detected by the bed detection unit 3, and the posture of the patient detected by the human posture detection unit 4. At this time, the distance from the calculated position of the center of gravity Gb of bed D1 to the ends of the reference rectangle D2 is calculated for each of the top, bottom, left, and right sides, and the distance from the center of gravity Gb of bed D1 to the reference rectangle D2 is set to 1, and the coordinates from the center of gravity Gb of bed D1 to the individual coordinates of the patient are expressed as a ratio of the distance to the reference rectangle D2 (relative coordinate value). In other words, the dimension from the center of gravity Gb of bed D1 to a specific bed end is set to 1, and the distance from the center of gravity Gb of bed D1 is calculated as a relative dimension based on this 1, and the patient's position is estimated.

[0021] The relative position information storage unit 6 stores the relative position information at a plurality of times calculated by the relative position calculation unit 5 in chronological order.

[0022] Fig. 3(a) shows the images captured by camera 1 in chronological order, and (b) shows the calculated bed D1 and patient. Five images F1 to F5 in Fig. 3(a) are shown in chronological order starting from F1, and Fig. 3(b) shows patient D3 in a triangular shape consisting of the coordinates of three points, the neck, left pelvis, and right pelvis, relative to bed D1. The coordinates calculated from the image F1 in Fig. 3(b) are only the coordinates of bed D1 and do not recognize the patient because he is lying down, but the coordinates of patient D3 corresponding to the image F2 recognize three points, the neck, right pelvis, and left pelvis, as he has raised his upper body, and calculate the state in which the patient is sitting up. After that, the coordinates corresponding to the images F3 to F5 show that the state in which patient D3 moves to the corner of bed D1 and then away from bed D1 as it moves from F3 to F5 is calculated. This information is stored as relative position information in the relative position information storage unit 6.

[0023] The behavior estimation unit 7 detects the patient's bed-getting behavior by inputting relative position information at multiple times to a neural network model LSTM (Long short-term memory). Information from the relative position information storage unit 6 is input to this neural network model LSTM, and the patient's bed-getting behavior is determined (estimated). Figure 4 is an explanatory diagram of the neural network, and the neural network structure inputs the three-point coordinates (6 units) of a person converted into relative coordinates for a fixed frame length of k steps, and connects L units of LSTM to the hidden layer. During learning, the transmission to the LSTM output value is stochastically controlled by dropout. Next, the output layer of the neural network calculates and outputs the likelihood using a softmax function as a two-class classification problem between getting out of bed and other actions, and it is determined whether the patient has gotten out of bed.

[0024] In this way, the patient's behavior can be estimated by observing the patient's posture on the bed and the changes in the relative position between the bed and the patient. In other words, if it is detected that the patient being monitored gets up on the bed and then sits down on the bed, the system will determine that anyone moving away from the bed after that is the patient leaving the bed. This reduces the chance of erroneous decisions being made that the leaving action of a person not being monitored, such as a nurse, is the patient's action. In addition, by using a softmax function, the patient's bed exit motion can be estimated with high accuracy, and by estimating the posture based on the patient's skeletal information, the posture can be estimated with high accuracy. [Explanation of symbols]

[0025] Reference Signs List 1: camera, 2: image storage unit, 3: bed detection unit, 4: human posture detection unit, 5: relative position calculation unit, 6: relative position information storage unit, 7: behavior estimation unit.

Claims

1. A camera that captures the entire surveillance area; A bed detection unit that extracts a bed from an image captured by the camera and calculates an area of ​​the bed from the extracted information; a person posture detection unit that obtains coordinate information of a specific part of the person on the bed from the image captured by the camera and detects the posture of the person from the information; a relative position calculation unit that calculates a center of gravity position of the bed based on information detected by the bed detection unit and the person posture detection unit and calculates a relative position between the bed and the person; a relative position information storage unit that stores the relative position information calculated by the relative position calculation unit in a time series; a behavior estimation unit that estimates a behavior of a person on the bed using a neural network based on the plurality of pieces of time-series information stored in the relative position information storage unit, The specific motion detection device is characterized in that the relative position calculation unit calculates a rectangular reference rectangle based on the diamond-shaped bed information detected by the bed detection unit from the image captured by the camera, and calculates the relative position of the patient from the reference rectangle, the center of gravity position of the bed, and the patient's posture detected by the human posture detection unit.

2. 2. The specific motion detection device according to claim 1, wherein the human posture detection unit obtains coordinate information of at least three points of the neck, left pelvis, and right pelvis out of the head, neck, left hand, right hand, left pelvis, and right pelvis of the person on the bed, and estimates the posture based on the coordinate information.

Images