Watching system, watching device, watching method, and watching program

Abstract

In order to efficiently detect a change in the physical condition of a resident, a watching system 10 captures an image of a resident by an imaging device 20, performs skeleton recognition on the basis of the image by a walking speed calculation device 30, and acquires walking trajectory data by acquiring chest coordinates in time series. The walking speed calculation device 30 calculates walking speeds on the basis of the walking trajectory data. A management device 40 cumulatively adds each calculated walking speed to generate a frequency distribution, and identifies the walking speed of the resident in a predetermined period on the basis of the frequency distribution. The management device 40 calculates a difference between the acquired walking speed and the identified walking speed, and determines whether the difference is equal to or greater than a predetermined threshold. When the difference is equal to or greater than the predetermined threshold, a notification is issued. A portable terminal 50 receives the result of the determination, and issues a notification when the difference is equal to or greater than the predetermined threshold.

Description

Monitoring system, monitoring device, monitoring method, and monitoring program

[0001] The present disclosure relates to a monitoring system, a monitoring device, a monitoring method, and a monitoring program that efficiently detect changes in the physical condition of a resident.

[0002] Conventionally, in nursing care facilities and the like, the walking speed of residents is calculated and the calculated walking speed is used as one of the indices for determining the level of care required. For this reason, a technique for determining the walking speed of residents in nursing care facilities and the like is known.

[0003] For example, Patent Document 1 discloses a technology for calculating walking speed from walking trajectory data of a resident and normalizing the walking speed based on the resident's attributes (age, sex, height, weight, etc.). Patent Document 2 discloses a technology for acquiring multiple trajectory data representing walking trajectories in a room on different days, distinguishing trajectory data of the resident from trajectory data of persons other than the resident, and calculating walking speed using only the trajectory data of the resident.

[0004] International Publication No. 2020-003954 Patent No. 7215481

[0005] However, although the above-mentioned Patent Document 1 normalizes walking speed based on the attributes of the resident, the attributes of the resident do not necessarily affect walking speed, as elderly women are often younger than elderly men. Therefore, the normalized walking speed may not reflect the actual situation of the resident, and it may be difficult to detect changes in the resident's physical condition.

[0006] In the above-mentioned Cited Document 2, after determining whether the walking trajectory data belongs to a resident or not, the walking speed is calculated using only the walking trajectory data determined to belong to a resident. However, since the determination is made based on the start and end points of the trajectory or the time period, the walking trajectory does not necessarily correspond to the walking trajectory of only the resident. As a result, there is a possibility that changes in the physical condition of the resident may not be detected properly.

[0007] The present disclosure has been made in consideration of the above-mentioned conventional problems, and aims to provide a monitoring system, monitoring device, monitoring method, and monitoring program that efficiently detect changes in the physical condition of residents.

[0008] The monitoring system according to the present disclosure is a monitoring system for monitoring a resident in a room, and includes a walking trajectory data acquisition unit that acquires walking trajectory data indicating the walking trajectory of a person in the room, a walking speed calculation unit that calculates the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation unit that accumulates the walking speeds for each speed to generate a frequency distribution, and a walking speed identification unit that identifies the walking speed of the resident based on the frequency distribution.

[0009] In the aforementioned configuration, the walking speed identification unit may identify the walking speed that is most frequently found in the frequency distribution as the resident walking speed.

[0010] In the aforementioned configuration, the predetermined length may be a predetermined measurement unit length of the walking trajectory that is determined in advance.

[0011] In the above configuration, the walking trajectory data acquisition unit may acquire the walking trajectory data by estimating a person's skeleton from a captured image and acquiring chest coordinates of the person in time series.

[0012] The above configuration may further include a memory unit that stores the resident walking speed, and an alarm unit that issues an alarm when the difference between the resident walking speed stored in the memory unit and the resident walking speed identified by the walking speed identification unit becomes equal to or greater than a predetermined threshold.

[0013] In the aforementioned configuration, the frequency distribution generating unit may generate a frequency distribution for a predetermined period of time, and the storage unit may store the resident walking speed for the predetermined period of time.

[0014] The above configuration may further include a determination unit that determines whether the walking trajectory data acquired by the walking trajectory data acquisition unit is suitable as a target for generating the frequency distribution, and the walking speed calculation unit may calculate the walking speed when the determination unit determines that the walking trajectory data is suitable as a target for generating the frequency distribution.

[0015] In the aforementioned configuration, when the walking trajectory acquired by the walking trajectory data acquisition unit is a walking trajectory of a plurality of people, the determination unit may determine that the walking trajectory is not suitable as a target for generating the frequency distribution.

[0016] In the aforementioned configuration, the determination unit may determine that the walking trajectory acquired by the walking trajectory data acquisition unit is not suitable as a target for generating the frequency distribution when the walking trajectory is present within a predetermined area set in advance.

[0017] In the above configuration, the determination unit may determine that the walking trajectory data is not suitable for generating the frequency distribution when a predetermined percentage or more of the walking trajectory data exists within a circle having its center at the start point and the end point of the walking trajectory of a measurement unit length and having a radius of a predetermined value less than the measurement unit length.

[0018] In the aforementioned configuration, the predetermined length may be a length of a line segment between a start point and an end point of a walking trajectory in the walking trajectory data, which is expressed as a regression line.

[0019] The monitoring device according to the present disclosure is a monitoring device that monitors a resident in a room, and includes a walking trajectory data acquisition unit that acquires walking trajectory data indicating the walking trajectory of a person in the room, a walking speed calculation unit that calculates the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation unit that accumulates the walking speeds for each speed to generate a frequency distribution, and a walking speed identification unit that identifies the walking speed of the resident based on the frequency distribution.

[0020] The monitoring method according to the present disclosure is a monitoring method executed by a monitoring system that monitors a resident in a room, and includes a walking trajectory data acquisition step of acquiring walking trajectory data indicating the walking trajectory of a person in the room, a walking speed calculation step of calculating the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation step of cumulatively adding the walking speeds for each speed to generate a frequency distribution, and a walking speed identification step of identifying the walking speed of the resident based on the frequency distribution.

[0021] The monitoring program according to the present disclosure is a monitoring program executed by a monitoring device that monitors a resident in a room, and causes a computer to execute the following steps: a walking trajectory data acquisition step to acquire walking trajectory data indicating the walking trajectory of a person in the room; a walking speed calculation step to calculate the walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generation step to generate a frequency distribution by cumulatively adding the walking speeds for each speed; and a walking speed identification step to identify the walking speed of the resident based on the frequency distribution.

[0022] According to the present disclosure, changes in the physical condition of a resident can be detected efficiently.

[0023] FIG. 1 is a diagram illustrating an overview of a monitoring system according to this embodiment. FIG. 2 is a diagram illustrating the system configuration of the monitoring system illustrated in FIG. 1. FIG. 3 is a functional block diagram illustrating the configuration of an imaging device illustrated in FIG. 2. FIG. 4 is a functional block diagram illustrating the configuration of a walking speed calculation device illustrated in FIG. 2. FIG. 5 is a functional block diagram illustrating the configuration of a management device illustrated in FIG. 2. FIG. 6 is a functional block diagram illustrating the configuration of a mobile terminal illustrated in FIG. 2. FIG. 7 is an explanatory diagram illustrating generation of a trained model. FIG. 8 is an explanatory diagram illustrating an excluded region that is not treated as walking trajectory data. FIG. 9 is an explanatory diagram illustrating calculation of chest coordinates. FIG. 10 is an explanatory diagram illustrating an example of a walking trajectory and calculation of walking speed. FIG. 11 is a diagram illustrating an example of walking trajectory data that is not suitable for generating a frequency distribution. FIG. 12 is a diagram illustrating an example of a frequency distribution of walking speeds. FIG. 13 is a sequence diagram illustrating a processing procedure for identifying a resident's walking speed in the monitoring system of FIG. 2. FIG. 14 is a flowchart illustrating the processing procedure for image acquisition processing of FIG. 13. FIG. 15 is a flowchart illustrating the processing procedure for walking trajectory data generation processing of FIG. 13. Fig. 16 is a flowchart showing the processing procedure for determining whether walking frequency data is appropriate in Fig. 13. Fig. 17 is a sequence diagram (part 1) showing the processing procedure for the monitoring system in Fig. 2. Fig. 18 is a sequence diagram (part 2) showing the processing procedure for the monitoring system in Fig. 2. Fig. 19 is a diagram showing an example of walking trajectory data of a monitoring system according to a modified example. Fig. 20 is an explanatory diagram explaining detection of a change point in walking direction in a monitoring system according to a modified example. Fig. 21 is a diagram showing an example of a hardware configuration.

[0024] Hereinafter, embodiments of a monitoring system, a monitoring device, a monitoring method, and a monitoring program according to the present disclosure will be described in detail with reference to the accompanying drawings.

[0025] [Overview of Monitoring System] An overview of the monitoring system according to this embodiment will be described. FIG. 1 is a diagram showing an overview of the monitoring system according to this embodiment. As shown in FIG. 1( a), the monitoring system includes an imaging device 20, a walking speed calculation device 30, a management device 40, and a mobile terminal 50. The imaging device 20 is installed on the ceiling of a room 90 in which a care-requiring person resides, captures an image of the room in the direction of a floor 91, and transmits the image to the walking speed calculation device 30. As shown in FIG. 1( b), the monitoring system calculates the walking speed of the resident based on the captured image 100, and accumulates the walking speeds for each speed to generate a frequency distribution. The monitoring system identifies the most frequent value in the frequency distribution as the walking speed of the resident and detects changes in the resident's physical condition based on the identified walking speed.

[0026] The walking speed calculation device 30 recognizes the skeleton of the resident in the room based on the image data transmitted from the imaging device 20 and acquires walking trajectory data of the resident from the chest coordinates. The walking speed calculation device 30 calculates the walking speed of the resident based on the walking trajectory data. The walking speed calculation device 30 transmits the walking speed of the resident to the management device 40. As will be described in detail later, if the walking trajectory data is in an area where a bed 92 is installed or an area close to a wall, the walking speed calculation device 30 performs processing to invalidate the walking trajectory data.

[0027] The management device 40 receives the walking speeds of the residents from the walking speed calculation device 30, and accumulates the walking speeds of the residents for each speed to generate a frequency distribution. The management device 40 identifies the walking speed of the residents based on the frequency distribution for a predetermined period. Specifically, for example, the management device 40 generates a cumulative distribution of the walking speeds of the residents for each speed for one month, and identifies the walking speed with the highest frequency in the cumulative distribution data as the walking speed of the residents.

[0028] When the management device 40 receives the walking speed of the resident from the walking speed calculation device 30, it compares the received walking speed of the resident with the identified walking speed of the resident and determines whether the difference is equal to or greater than a predetermined threshold. If the difference is equal to or greater than the threshold, the management device 40 issues an alarm. The management device 40 transmits walking speed determination result data to the mobile terminal 50. The mobile terminal 50 determines whether to issue an alarm based on the determination result. If the determination result is "problem," the mobile terminal 50 issues an alarm.

[0029] [System Configuration of Monitoring System 10] The system configuration of the monitoring system 10 will be described. Fig. 2 is a diagram showing the system configuration of the monitoring system 10 shown in Fig. 1. As shown in Fig. 2, the monitoring system 10 has an imaging device 20, a walking speed calculation device 30, a management device 40, and a mobile terminal 50. The walking speed calculation device 30, the management device 40, and the mobile terminal 50 are connected via a network N. The imaging device 20 is connected to the walking speed calculation device 30.

[0030] The imaging device 20 is a device that captures images of residents in the living room. The imaging device 20 has a motor that drives an imaging unit, recognizes people based on images captured by the imaging unit, and controls the motor so that the people are included in the angle of view of the image data.

[0031] The walking speed calculation device 30 performs a skeletal structure recognition process based on the image data transmitted from the imaging device 20, and acquires walking trajectory data of the resident. The walking speed calculation device 30 is a device that determines whether the acquired walking trajectory data is suitable for calculating a walking speed, and when it is determined that the walking trajectory data is suitable for calculating a walking speed, performs a process of calculating the walking speed based on the walking trajectory data.

[0032] The management device 40 performs a process of generating a cumulative distribution for each speed based on the walking speed data of the resident transmitted from the walking speed calculation device 30, and a process of identifying the resident's walking speed based on the cumulative distribution data of walking speeds for a predetermined period. The management device 40 compares the walking speed of the resident transmitted from the walking speed calculation device 30 with the identified resident walking speed, and determines whether or not the difference between the resident's walking speed and the identified resident walking speed is equal to or greater than a predetermined threshold. If the difference is equal to or greater than the predetermined threshold, the management device 40 issues an alarm and transmits to the mobile terminal 50 data indicating the determination result as to whether or not the difference between the resident's walking speed and the identified resident walking speed is equal to or greater than the predetermined threshold.

[0033] The mobile terminal 50 is a device that receives the judgment result data from the management device 40 and performs processing to display the status of the resident based on the judgment result data. If the judgment result data indicates "problem," the mobile terminal 50 performs processing to issue an alarm.

[0034] [Configuration of Image Capturing Device 20] The configuration of the image capturing device 20 shown in Fig. 2 will be described. Fig. 3 is a functional block diagram showing the configuration of the image capturing device 20 shown in Fig. 2. As shown in Fig. 3, the image capturing device 20 has an image capturing unit 21, a light source unit 22, a motor 23, a communication I / F unit 24, a storage unit 25, and a control unit 26. The image capturing unit 21 is a camera that captures images of a resident in a room. Specifically, the image capturing unit 21 acquires infrared images for skeletal structure recognition in the walking speed calculation device 30, which will be described later. The light source unit 22 is a device that irradiates infrared light to capture infrared images.

[0035] The motor 23 is a drive unit used to change the angle of view of the imaging unit 21. A DC motor, a servo motor, a stepping motor, or the like is used as the motor 23. The motor 23 is controlled by a motor control unit 26d, which will be described later. The communication I / F unit 24 is an interface unit for communicating with the walking speed calculation device 30, and performs wireless communication using mobile communication standards such as third-generation mobile communication standards, fourth-generation mobile communication standards, and fifth-generation mobile communication standards, as well as short-range wireless communication using a wireless LAN (Local Area Network) or the like.

[0036] The storage unit 25 is a storage device such as a hard disk drive or nonvolatile memory, and stores a person recognition trained model 25a and image data 25b. The person recognition trained model 25a is a trained model downloaded from a server device (not shown). Specifically, as shown in FIG. 7 , training data A consisting of a large amount of data is input to a CNN (Convolutional Neural Network) B, and backpropagation is performed based on correct answer data R, for example, and supervised learning that determines the weight of each path is repeated to generate a trained model M. Note that the correct answer data R, i.e., the probability that it is a known phenomenon, can be assigned by the person performing the supervised learning. The image data 25b is image data that combines an infrared image captured by the imaging unit 21 with depth information.

[0037] The control unit 26 is a control unit that controls the entire imaging device 20, and includes a light source control unit 26a, an image acquisition unit 26b, a person recognition unit 26c, a motor control unit 26d, and an image transmission unit 26e. In practice, these programs are loaded into a CPU, a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), or the like and executed, causing the light source control unit 26a, the image acquisition unit 26b, the person recognition unit 26c, the motor control unit 26d, and the image transmission unit 26e to execute the processes corresponding thereto.

[0038] The light source control unit 26a performs on / off control of the light source unit 22. The image acquisition unit 26b acquires infrared images using the imaging unit 21 and acquires depth information using ToF (Time of Flight) technology. The person recognition unit 26c inputs the captured images into the person recognition trained model 25a and recognizes people.

[0039] The motor control unit 26d controls the motor 23 to adjust the angle of view of the imaging unit 21. The image transmission unit 26e transmits the captured image data 25b to the walking speed calculation device 30 via the communication I / F unit 24.

[0040] [Configuration of walking speed calculation device 30] The configuration of the walking speed calculation device 30 will be described. Fig. 4 is a functional block diagram showing the configuration of the walking speed calculation device 30 shown in Fig. 2. As shown in Fig. 4, the walking speed calculation device 30 has a communication I / F unit 31, a storage unit 34, and a control unit 35. The communication I / F unit 31 is an interface unit for communicating with the imaging device 20 and the management device 40, and performs wireless communication using mobile communication standards such as a third-generation mobile communication system, a fourth-generation mobile communication system, or a fifth-generation mobile communication system, or short-range wireless communication using a wireless LAN (Local Area Network) or the like.

[0041] The storage unit 34 is a storage device such as a hard disk drive or non-volatile memory, and stores a skeleton recognition trained model 34a, image data 34b, skeleton recognition data 34c, walking trajectory data 34d, and excluded area data 34e. The skeleton recognition trained model 34a is a trained model downloaded from a server device (not shown) or the like. The skeleton recognition trained model 34a is a trained model that inputs image data 34b and recognizes the skeleton of a person included in the image. Note that the generation of the trained model has already been explained, so a detailed explanation will be omitted here.

[0042] The image data 34b is data of images and depth information received from the imaging device 20. The skeleton recognition data 34c is data obtained by performing skeleton recognition on the person included in the image data 34b. The walking trajectory data 34d is data of the trajectory of the resident's walking. The excluded area data 34e is data representing areas of the location where the resident has walked that are not recognized as a walking trajectory. Specifically, as shown in FIG. 8 , on the floor 91 of a room 90, a bed area 94 where a bed 92 is installed, a wall area 95, etc. are set as excluded areas. Note that the excluded area may include areas that are set because the resident's normal walking speed cannot be obtained, depth information is unreliable due to the camera's performance, etc.

[0043] The control unit 35 is a control unit that controls the entire walking speed calculation device 30, and has an image data receiving unit 35a, a skeleton recognition unit 35b, a walking trajectory data acquisition unit 35c, a determination unit 35d, a walking speed calculation unit 35e, and a walking speed data transmission unit 35f. In practice, by loading and executing these programs into the CPU, the image data receiving unit 35a, the skeleton recognition unit 35b, the walking trajectory data acquisition unit 35c, the determination unit 35d, the walking speed calculation unit 35e, and the walking speed data transmission unit 35f execute processes corresponding to them, respectively.

[0044] The image data receiving unit 35a receives image data transmitted from the imaging device 20 and stores the image data in the storage unit 34. The skeleton recognition unit 35b performs skeleton recognition of a person included in the image data 34b based on the image data 34b. Specifically, as shown in Fig. 9, skeleton recognition of a person 111 included in an image 110 received from the imaging device 20 is performed by inputting the image 110 to the skeleton recognition trained model 24a and acquiring a skeleton model 130 of the person 111.

[0045] Here, the skeletal model 130 is recognized by the following parts: mouth: B0, chest: B1, right shoulder: B2, right elbow: B3, right wrist: B4, left shoulder: B5, left elbow: B6, left wrist: B7, right groin: B8, right knee: B9, right ankle: B10, left groin: B11, left knee: B12, left ankle: B13, right eye: B14, left eye: B15, right ear: B16, left eye: B17.

[0046] Thereafter, the skeleton recognition unit 35 b identifies the three-dimensional coordinates 140 of each part of the skeleton model 130 based on the recognized skeleton model 130 and the depth information 120 .

[0047] The walking trajectory data acquisition unit 35c acquires walking trajectory data 34d by acquiring chest coordinates of the person whose skeleton has been recognized in time series. The walking trajectory data acquisition unit 35c acquires data for each predetermined measurement unit length. Note that the predetermined measurement unit length is, for example, 2 m, and when the movement distance of the time series of chest coordinates whose skeleton has been recognized reaches 2 m, the data is stored in the storage unit 34 as one piece of walking trajectory data 34d.

[0048] Specifically, as shown in FIG. 10(a), the chest coordinates of the skeleton-recognized person are acquired in time series, and the amount of change in the chest coordinates is cumulatively added. When the cumulative value reaches a predetermined measurement unit length, the walking start W is calculated as shown in FIG. Ｓ And finished walking. Ｅ The straight line connecting these points is stored in the storage unit 34 as walking trajectory data 34d.

[0049] The determination unit 35d determines whether or not the walking speed calculated based on the acquired walking trajectory data 34d is suitable for use as a target for generating a frequency distribution, which will be described later, in the management device 40. Specifically, the determination unit 35d determines that the walking trajectory data 34d is inappropriate if the walking trajectories of multiple people are recorded in the walking trajectory data 34d or if the walking trajectory data 34d includes coordinates set in the exclusion area data 34e.

[0050] As shown in FIG. 11, the determination unit 35d determines the walking start W of a predetermined measurement unit length. Ｓ And finished walking. Ｅ The walking trajectory data 34d is judged to be appropriate based on whether a predetermined ratio of multiple chest coordinates exists within a circle having a radius of a predetermined value less than a predetermined measurement unit length, with each of the chest coordinates being centered at a predetermined value. For example, if the predetermined ratio is set to 60%, as shown in FIG. 11(a), Ｓ And finished walking. Ｅ There are multiple chest coordinates within a circle of a predetermined radius with each of the coordinates at the center, but the proportion is less than 60%, so this walking trajectory data 34d is determined to be appropriate, and as shown in FIG. 11(b), Ｓ And finished walking. Ｅ If 60% or more of the chest coordinates are within a circle of a predetermined radius centered on each of the points, the walking trajectory data 34d is determined to be inappropriate.

[0051] The walking speed calculation unit 35e calculates the walking speed based on the walking trajectory data 34d and the time required to walk a predetermined measurement unit length. The walking speed data transmission unit 35f transmits the calculated walking speed data to the management device 40.

[0052] [Configuration of management device 40] The configuration of the management device 40 will be described. Fig. 5 is a functional block diagram showing the configuration of the management device 40 shown in Fig. 2. As shown in Fig. 5, the management device 40 has a display unit 41, an input unit 42, a communication I / F unit 43, a storage unit 44, and a control unit 45. The display unit 41 is a display device such as a liquid crystal display that displays various information.

[0053] The input unit 42 is an input device such as a mouse, a keyboard, etc. The communication I / F unit 43 is an interface unit for communicating with the walking speed calculation device 30 and the mobile terminal 50, and performs wireless communication using mobile communication standards such as a third-generation mobile communication system, a fourth-generation mobile communication system, and a fifth-generation mobile communication system, and short-range wireless communication using a wireless LAN (Local Area Network) or the like.

[0054] The storage unit 34 is a storage device such as a hard disk drive or nonvolatile memory, and stores walking speed data 44a, walking speed frequency distribution data 44b, specific resident walking speed data 44c, and judgment result data 44d. The walking speed data 44a is data on the walking speed of the resident received from the walking speed calculation device 30. The walking speed frequency distribution data 44b is data obtained by accumulating the walking speed for each speed.

[0055] The specific resident walking speed data 44c is data that identifies the most frequent walking speed based on the walking speed frequency distribution data 44b for a predetermined period. The judgment result data 44d is data of the judgment result obtained by comparing the resident walking speed data 44a with the specific resident walking speed data 44c of the resident and determining whether the difference is within a predetermined threshold.

[0056] The control unit 45 is a control unit that controls the entire management device 40, and has a walking speed data receiving unit 45a, a frequency distribution generating unit 45b, a walking speed specifying unit 45c, a walking speed determining unit 45d, a notifying unit 45e, and a determination result data transmitting unit 45f. In practice, by loading these programs into a CPU and executing them, the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the determination result data transmitting unit 45f will execute processes corresponding to them, respectively.

[0057] The walking speed data receiving unit 45a receives the walking speed data of the resident calculated by the walking speed calculation device 30 and stores it as walking speed data 44a in the storage unit 44. The frequency distribution generating unit 45b cumulatively adds up the walking speed data 44a of the resident for each walking speed.

[0058] The walking speed identification unit 45c identifies the walking speed of the resident for a predetermined period based on the walking speed frequency distribution data 44b, and stores the identified resident walking speed data 44c in the storage unit 44. For example, as shown in Fig. 12, walking speeds are classified in increments of 0.05 m / s within a range from 0 to 1.0 m / s, and the most frequently occurring walking speed of 0.25 m / s to 0.30 m / s is identified as the resident's walking speed for the predetermined period based on the cumulative distribution data for each speed.

[0059] The walking speed determination unit 45d compares the walking speed data 44a transmitted from the walking speed calculation device 30 with the specific resident walking speed data 44c for the predetermined period identified by the walking speed identification unit 45c, and determines whether the difference is equal to or greater than a predetermined threshold. If the difference between the walking speed data 44a transmitted from the walking speed calculation device 30 and the specific resident walking speed data 44c for the predetermined period identified by the walking speed identification unit 45c is within the predetermined threshold, the walking speed determination unit 45d determines that the walking speed is "no problem," and if the difference is greater than the predetermined threshold, the walking speed determination unit 45d determines that the walking speed is "problematic." The walking speed determination unit 45d then stores the determination result data 44d in the storage unit 44.

[0060] The notification unit 45e issues a notification based on the judgment result data 44d. Specifically, if the judgment result data 44d is "problem", the notification unit 45e displays the walking speed on the predetermined display unit 41 and sounds an alarm. On the other hand, if the judgment result data 44d is "no problem", the notification unit 45e displays the walking speed on the predetermined display unit 41.

[0061] The determination result data transmitting unit 45 f transmits the determination result data 44 d to the mobile terminal 50 .

[0062] [Configuration of portable terminal 50] The configuration of portable terminal 50 will be described. Fig. 6 is a functional block diagram showing the configuration of portable terminal 50 shown in Fig. 2. As shown in Fig. 6, portable terminal 50 has an operation display unit 51, a communication I / F unit 52, a storage unit 54, and a control unit 55. Operation display unit 51 is an input and display device such as a touch panel liquid crystal display.

[0063] The communication I / F unit 52 is an interface unit that communicates with the management device 40, and performs wireless communication using mobile communication standards such as third-generation mobile communication systems, fourth-generation mobile communication systems, and fifth-generation mobile communication systems, as well as short-range wireless communication using a wireless LAN (Local Area Network), etc. The storage unit 54 is a storage device such as a hard disk drive or nonvolatile memory.

[0064] The control unit 55 is a control unit that controls the entire portable terminal 50, and has a determination result data receiving unit 55a and a notification unit 55b. In practice, by loading these programs into the CPU and executing them, the determination result data receiving unit 55a and the notification unit 55b will execute processes corresponding to them, respectively.

[0065] The judgment result data receiving unit 55a receives the judgment result data 44d transmitted from the management device 40. The notification unit 55b issues a notification based on the judgment result data 44d. Specifically, if the judgment result data 44d indicates "problem", the notification unit 55b displays the walking speed on the predetermined operation display unit 51 and sounds an alarm. Furthermore, if the judgment result data 44d indicates "no problem", the notification unit 55b displays the walking speed on the predetermined operation display unit 51.

[0066] [Processing Procedure for Identifying Resident Walking Speed ​​in Monitoring System] A processing procedure for identifying the resident walking speed in the monitoring system 10 will be described. Fig. 13 is a sequence diagram showing the processing procedure for identifying the resident walking speed in the monitoring system 10 of Fig. 2. As shown in Fig. 13, the imaging device 20 of the monitoring system 10 performs an image acquisition process (step S101). The imaging device 20 transmits image data to the walking speed calculation device 30 (step S102).

[0067] The walking speed calculation device 30 determines whether or not image data has been received from the imaging device 20 (step S103). If image data has not been received, the walking speed calculation device 30 waits until image data is received (step S103: No).

[0068] If the walking speed calculation device 30 receives the image data (step S103: Yes), it performs a walking trajectory data generation process (step S104). The walking speed calculation device 30 determines whether the resident's movement distance exceeds a predetermined threshold (measurement unit length) (step S105). If the movement distance does not exceed the predetermined threshold (step S105: No), the walking speed calculation device 30 proceeds to step S103 and waits for reception of image data from the imaging device 20.

[0069] On the other hand, if the movement distance exceeds the predetermined threshold (step S105: Yes), the walking speed calculation device 30 performs a process of determining whether the walking trajectory data is appropriate (step S106). The walking speed calculation device 30 determines whether the walking trajectory data is appropriate as data for calculating a walking speed and accumulating the frequency distribution data (step S107). If the walking trajectory data is not appropriate (step S107: No), the walking speed calculation device 30 proceeds to step S103 and waits for image data to be transmitted from the imaging device 20.

[0070] On the other hand, if the walking trajectory data is appropriate (step S107: Yes), the walking speed calculation device 30 calculates the walking speed (step S108) and transmits the walking speed data to the management device 40 (step S109).

[0071] The management device 40 determines whether or not walking speed data has been received from the walking speed calculation device 30 (step S110). If the management device 40 has not received walking speed data (step S110: No), the management device 40 waits for reception of walking speed data. On the other hand, if the management device 40 has received walking speed data from the walking speed calculation device 30 (step S110: Yes), the management device 40 cumulatively adds the received walking speed data to the walking speed frequency distribution data for each speed (step S111).

[0072] The management device 40 determines whether a predetermined period of time has elapsed (step S112). If the predetermined period of time has not elapsed (step S112: No), the management device 40 terminates the process. On the other hand, if the predetermined period of time has elapsed (step S112: Yes), the management device 40 identifies the most frequently occurring walking speed as the resident walking speed based on the walking speed frequency distribution data (step S113).

[0073] [Processing Procedure of Image Acquisition Processing] The processing procedure of the image acquisition processing shown in Fig. 13 will be described. Fig. 14 is a flowchart showing the processing procedure of the image acquisition processing shown in Fig. 13. The imaging device 20 captures an image of the living room 90 using the imaging unit 21 (step S201). The imaging device 20 recognizes a person from the captured image data (step S202).

[0074] The imaging device 20 adjusts the angle of the imaging unit 21 so that the person is within the imaging range (step S203). The imaging device 20 captures an image of the living room 90 using the imaging unit 21 (step S204), and the process returns to step S102 in FIG.

[0075] [Processing Procedure for Walking Trajectory Data Generation Processing] The processing procedure for the walking trajectory data generation processing shown in Fig. 13 will be described. Fig. 15 is a flowchart showing the processing procedure for the walking trajectory data generation processing shown in Fig. 13. As shown in Fig. 15, the walking speed calculation device 30 performs skeleton recognition based on image data (step S301).

[0076] The walking speed calculation device 30 acquires chest coordinates based on the skeleton recognition data (step S302). The walking speed calculation device 30 stores the chest coordinates as a walking trajectory (step S303). The walking speed calculation device 30 calculates the movement distance from the difference in chest coordinates (step S304), and returns to step S105 in FIG. 13.

[0077] [Processing Procedure for Determining Walking Frequency Data Appropriateness] The processing procedure for determining whether walking frequency data is appropriate will be described with reference to Fig. 13. Fig. 16 is a flowchart showing the processing procedure for determining whether walking frequency data is appropriate with reference to Fig. 13. As shown in Fig. 16, the walking speed calculation device 30 determines whether the walking trajectory data 34d includes one or more people (step S401). If no one is captured (step S401: No), the walking speed calculation device 30 determines that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in Fig. 13.

[0078] On the other hand, if the number of people shown is one or more (step S401: Yes), the walking speed calculation device 30 determines whether there is one moving person (step S402).If there are two or more moving people (step S402: No), the walking speed calculation device 30 determines that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG.

[0079] On the other hand, if there is only one moving person (step S402: Yes), the walking speed calculation device 30 determines whether the chest coordinates are outside the exclusion area (step S403). If the chest coordinates are within the exclusion area (step S403: No), the walking speed calculation device 30 determines that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG. 13 .

[0080] On the other hand, if the chest coordinate is outside the exclusion area (step S403: Yes), the walking speed calculation device 30 determines whether the up and down movement of the chest coordinate is equal to or less than a predetermined threshold (step S404). If the up and down movement of the chest coordinate is greater than the predetermined threshold (step S404: No), the walking speed calculation device 30 determines that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG. 13 .

[0081] On the other hand, if the up-and-down movement of the chest coordinate is equal to or less than the predetermined threshold (step S404: Yes), the walking speed calculation device 30 determines whether the chest coordinate is present in a circle of a predetermined size centered on the start and end points of walking at a predetermined rate or more (step S405). If the chest coordinate is present in a circle of a predetermined size centered on the start and end points of walking at a predetermined rate or more (step S405: No), the walking speed calculation device 30 determines that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG. 13 .

[0082] On the other hand, if the chest coordinate does not exist at a rate equal to or greater than the predetermined rate (step S405: Yes), the walking speed calculation device 30 determines that the walking trajectory data 34d is "appropriate" (step S406), and returns to step S107 in FIG.

[0083] [Processing Procedure of Monitoring System 10] The processing procedure of the monitoring system 10 shown in Fig. 2 when monitoring a resident will be described. Figs. 17 and 18 are sequence diagrams showing the processing procedure of the monitoring system 10 of Fig. 2. As shown in Fig. 17, the imaging device 20 of the monitoring system 10 performs image acquisition processing (step S501). The imaging device 20 transmits image data to the walking speed calculation device 30 (step S502). Note that the image acquisition processing has already been described, so a detailed description will be omitted here.

[0084] The walking speed calculation device 30 determines whether or not image data has been received from the imaging device 20 (step S503). If image data has not been received, the walking speed calculation device 30 waits until image data is received (step S503: No).

[0085] If the walking speed calculation device 30 receives the image data (step S503: Yes), it performs a walking trajectory data generation process (step S504). The walking speed calculation device 30 determines whether the resident's movement distance exceeds a predetermined threshold (step S505). If the movement distance does not exceed the predetermined threshold (step S505: No), the walking speed calculation device 30 proceeds to step S503 and waits for reception of image data from the imaging device 20. Note that the walking trajectory data generation process has already been described, so a detailed description thereof will be omitted here.

[0086] On the other hand, if the movement distance exceeds a predetermined threshold (step S505: Yes), the walking speed calculation device 30 performs a walking trajectory data appropriateness determination process (step S506). The walking speed calculation device 30 determines whether the walking trajectory data is appropriate as data for calculating a walking speed and accumulating the frequency distribution data (step S507). If the walking trajectory data is not appropriate (step S507: No), the walking speed calculation device 30 proceeds to step S503 and waits for image data to be transmitted from the imaging device 20. Note that the walking trajectory data appropriateness determination process has already been described, so a detailed description thereof will be omitted here.

[0087] On the other hand, if the walking trajectory data is appropriate (step S507: Yes), the walking speed calculation device 30 calculates the walking speed (step S508) and transmits the walking speed data to the management device 40 (step S509).

[0088] The management device 40 determines whether walking speed data has been received from the walking speed calculation device 30 (step S510). If the management device 40 has not received walking speed data (step S510: No), it waits for reception of walking speed data. On the other hand, if the management device 40 has received walking speed data from the walking speed calculation device 30 (step S510: Yes), it reads out the specific resident walking speed data 44c identified by the walking speed identification unit 45c (step S511) and compares the walking speed data with the specific resident walking speed data 44c (step S512).

[0089] If the difference between the walking speed and the specific resident's walking speed is within a predetermined threshold (step S513: Yes), the management device 40 determines that the walking speed is "not problematic" (step S514). On the other hand, if the difference between the walking speed and the specific resident's walking speed is not within the predetermined threshold (step S513: No), the management device 40 determines that the walking speed is "problematic" (step S515).

[0090] The management device 40 transmits the determination result to the mobile terminal 50 (step S516) and displays the determination result data on the predetermined display unit 41 (step S517). The management device 40 determines whether the determination result is "problem-present" (step S518). If the determination result is "problem-present" (step S518: Yes), a warning is issued (step S519). The warning is issued by displaying a message such as "The walking speed of resident XX has decreased by 20%." On the other hand, if the determination result is "no problem" (step S518: No), the management device 40 ends the process.

[0091] When the portable terminal 50 receives the judgment result data (step S520), it displays the received judgment result data on the predetermined operation display unit 51 (step S521). The portable terminal 50 determines whether the judgment result is "problem" (step S522). If the judgment result is "problem" (step S522: Yes), it issues a warning (step S523). The warning is issued by displaying a message such as "The walking speed of resident XX has decreased by 20%." On the other hand, if the judgment result is "no problem" (step S522: No), the portable terminal 50 ends the process.

[0092] As described above, in this embodiment, the monitoring system 10 captures images of the resident using the imaging device 20, and the walking speed calculation device 30 performs skeletal recognition based on the images and acquires walking trajectory data by chronologically acquiring chest coordinates. The walking speed calculation device 30 then calculates the walking speed based on the walking trajectory data. The management device 40 accumulates the calculated walking speeds for each speed to generate a frequency distribution, and identifies the resident's walking speed for a predetermined period based on the frequency distribution. The management device 40 then calculates the difference between the acquired walking speed and the identified walking speed and determines whether the difference is equal to or greater than a predetermined threshold. If the difference is equal to or greater than the predetermined threshold, a notification is issued. The mobile terminal 50 also receives the determination result and issues a notification if the difference is equal to or greater than a predetermined threshold, thereby efficiently detecting changes in the resident's physical condition.

[0093] <Modification> In the monitoring system 10 according to the above embodiment, the walking path of the resident is Ｓ And finished walking. Ｅ In the above description, the case where the walking path of the resident is represented by a single regression line has been explained, but in the modified example, a case where the walking path of the resident is not represented by a single regression line will be explained.

[0094] 19A and 19B are diagrams showing an example of walking path data of a monitoring system according to a modified example. As shown in FIG. 19A, when the walking path of a resident draws multiple curves, a predetermined measurement unit length (for example, W Ｓ From W Ｅ Even if the distance is less than the distance to the Ｓ The first change point W where the walking direction of the resident's walking trajectory changes from ｎ Alternatively, a regression line may be calculated to calculate the walking speed based on the time required to travel this distance and the length L1 of the regression line.

[0095] Furthermore, as shown in FIG. 19(b), when the resident further changes the walking direction, the first change point W ｎ Starting from the second change point W ｍ Alternatively, a regression line may be calculated to calculate the walking speed based on the time required to travel this distance and the length L2 of the regression line.

[0096] Furthermore, as shown in FIG. 19(c), the monitoring system detects the second change point W ｍ After walking from Ｅ Alternatively, the walking speed may be calculated based on the time required to travel this distance and the length L3 of the regression line.

[0097] [Detection of a change point] The detection of a change point in the walking direction of a resident by the monitoring system according to the modified example will be described. FIG. 20 is an explanatory diagram for explaining the detection of a change point in the walking direction by the monitoring system according to the modified example. As shown in FIG. 20(a), when the resident starts walking, Ｓ With the origin as the origin, the coordinates of the time series data of the chest coordinates are respectively g ０ , g １ , g ２ Let's say.

[0098] As shown in FIG. 20(b), when the walking ｓ Starting from the origin, start walking W ｓ and the first chest coordinate time series data g ０ The angle on the XY plane formed by the and is the reference angle. ｓ and second chest coordinate time series data g １ The difference θ between the angle on the XY plane and the reference angle １ Furthermore, when the walking start W ｓ and the third chest coordinate time series data g ２ The difference θ between the angle on the XY plane and the reference angle ２ Calculate.

[0099] The monitoring system starts walking like this. ｓ With the origin as the time series data of the first chest coordinate, g ０ The angle on the XY plane formed by these coordinates is set as the reference angle. If the difference between the coordinates of the time-series data of each chest coordinate and the reference angle exceeds a predetermined threshold, it is determined that the resident has changed their walking direction.

[0100] In the above embodiment, the management device 40 of the monitoring system 10 is described as transmitting the judgment result data of the resident's walking speed to the mobile terminal 50, but the management device 40 may also be configured to transmit warning data to the mobile terminal 50 when the judgment result data indicates a "problem."

[0101] <Relationship with Hardware> The correspondence between the management device 40 of the monitoring system 10 according to this embodiment and the main hardware configuration of a computer will be described. Fig. 21 is a diagram showing an example of the hardware configuration.

[0102] Generally, a computer has a configuration in which a CPU 81, a ROM 82, a RAM 83, a non-volatile memory 84, etc. are connected via a bus 85. A hard disk drive may be provided instead of the non-volatile memory 84. For the sake of convenience of explanation, only the basic hardware configuration is shown.

[0103] Here, the ROM 82 or non-volatile memory 84 stores programs required to start the operating system (hereinafter simply referred to as "OS"), and the CPU 81 loads and executes the OS program from the ROM 82 or non-volatile memory 84 when the power is turned on.

[0104] On the other hand, various application programs executed on the OS are stored in non-volatile memory 84, and the CPU 81 executes the application programs while using RAM 83 as the main memory, thereby executing processes corresponding to the applications.

[0105] The monitoring program of the management device 40 of the monitoring system 10 according to this embodiment is stored in the non-volatile memory 84 or the like, like other application programs, and the CPU 81 loads and executes the monitoring program. In the case of the management device 40 of the monitoring system 10 according to this embodiment, a monitoring program including routines corresponding to the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the determination result data transmitting unit 45f shown in FIG. 5 is stored in the non-volatile memory 84 or the like. When the CPU 81 loads and executes the monitoring program, a monitoring process corresponding to the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the determination result data transmitting unit 45f is realized.

[0106] The configurations illustrated in the above embodiments are merely functional schematics and do not necessarily have to be physically configured as shown. In other words, the distribution and integration of each device is not limited to the illustrated configuration, and all or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Claims

1. A monitoring system for monitoring a resident in a room, comprising: a walking trajectory data acquisition unit that acquires walking trajectory data indicating a walking trajectory of a person in the room; a walking speed calculation unit that calculates the walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generation unit that accumulates the walking speeds for each speed to generate a frequency distribution; and a walking speed identification unit that identifies the walking speed of the resident based on the frequency distribution.

2. The monitoring system according to claim 1, characterized in that the walking speed identification unit identifies the walking speed that is most frequently found in the frequency distribution as the resident's walking speed.

3. The monitoring system according to claim 1, characterized in that the specified length is a predetermined measurement unit length of the walking trajectory that is determined in advance.

4. The monitoring system according to claim 1, characterized in that the walking trajectory data acquisition unit acquires the walking trajectory data by estimating the person's skeleton from a captured image and acquiring the chest coordinates in chronological order.

5. A monitoring system as described in any one of claims 1 to 4, further comprising: a memory unit that stores the resident walking speed; and an alarm unit that issues an alarm when the difference between the resident walking speed stored in the memory unit and the resident walking speed identified by the walking speed identification unit becomes equal to or greater than a predetermined threshold value.

6. The monitoring system according to claim 5, characterized in that the frequency distribution generating unit generates the frequency distribution for a predetermined period of time, and the memory unit stores the resident walking speed for the predetermined period of time.

7. The monitoring system described in claim 1, further comprising a judgment unit that judges whether the walking trajectory data acquired by the walking trajectory data acquisition unit is suitable as a target for generating the frequency distribution, and the walking speed calculation unit calculates the walking speed when the judgment unit judges that the walking trajectory data is suitable as a target for generating the frequency distribution.

8. The monitoring system according to claim 7, characterized in that the judgment unit judges that the walking trajectory acquired by the walking trajectory data acquisition unit is not suitable as a subject for generating the frequency distribution when it is the walking trajectory of multiple people.

9. The monitoring system of claim 7, characterized in that the judgment unit judges that the walking trajectory acquired by the walking trajectory data acquisition unit is not suitable as a subject for generating the frequency distribution when the walking trajectory is present within a predetermined specified area.

10. The monitoring system of claim 7, characterized in that the judgment unit judges that the walking trajectory data is not suitable for generating the frequency distribution when a predetermined percentage or more of the walking trajectory data is present within a circle whose center is the start point and end point of the walking trajectory of the measurement unit length and whose radius is a predetermined value less than the measurement unit length.

11. The monitoring system according to claim 1, characterized in that the specified length is the length of a line segment between a start point and an end point of a walking trajectory in the walking trajectory data expressed as a regression line.

12. A monitoring device for monitoring a resident in a room, comprising: a walking trajectory data acquisition unit for acquiring walking trajectory data indicating a walking trajectory of a person in the room; a walking speed calculation unit for calculating the walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generation unit for generating a frequency distribution by accumulating the walking speeds for each speed; and a walking speed identification unit for identifying the walking speed of the resident based on the frequency distribution.

13. A monitoring method executed by a monitoring system for monitoring a resident in a room, comprising: a walking trajectory data acquisition step of acquiring walking trajectory data indicating a walking trajectory of a person in the room; a walking speed calculation step of calculating the walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generation step of generating a frequency distribution by accumulating the walking speeds for each speed; and a walking speed identification step of identifying the walking speed of the resident based on the frequency distribution.

14. A monitoring program executed by a monitoring device that monitors a resident in a room, the monitoring program causing a computer to execute the following steps: a walking trajectory data acquisition step for acquiring walking trajectory data indicating a walking trajectory of a person in the room; a walking speed calculation step for calculating the walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generation step for generating a frequency distribution by accumulating the walking speeds for each speed; and a walking speed identification step for identifying the walking speed of the resident based on the frequency distribution.

Images