Monitoring system, monitoring device, monitoring method, and monitoring program

Abstract

To efficiently detect a change in a body condition of a resident.SOLUTION: A monitoring system 10 captures an image of a resident using an imaging device 20; a walking speed calculation device 30 performs skeleton recognition based on an image; and walking trajectory data is acquired by acquiring chest coordinates in a time sequential manner. Thereafter, the walking speed calculation device 30 calculates a walking speed based on the walking trajectory data. A management device 40 generates a frequency distribution by cumulatively adding the calculated walking speed for each speed, and specifies a walking speed of a resident in a predetermined period based on the frequency distribution. Thereafter, the management device 40 calculates a difference between the acquired walking speed and the specified walking speed, and determines whether the difference is equal to or greater than a predetermined threshold value, and performs a notification when the difference becomes equal to or greater than the predetermined threshold value. Also, a portable terminal 50 also receives a result of the determination, and if the difference becomes equal to or greater than the predetermined threshold value, issues a notification.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a monitoring system, a monitoring device, a monitoring method, and a monitoring program that efficiently detect changes in the physical condition of a resident. [Background technology]

[0002] Conventionally, in nursing care facilities and the like, the walking speed of residents is calculated, and the calculated walking speed is used as one of the indices for determining the level of care required. For this reason, a technique for determining the walking speed of residents in nursing care facilities and the like is known.

[0003] For example, Patent Document 1 discloses a technique for calculating walking speed from walking trajectory data of a resident and normalizing the walking speed based on the resident's attributes (age, sex, height, weight, etc.). Patent Document 2 discloses a technique for acquiring multiple trajectory data on different days showing walking trajectories in a room, distinguishing between trajectory data of non-residents and trajectory data of the resident, and calculating the walking speed using only the trajectory data of the resident. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020-003954 [Patent Document 2] Patent No. 7215481 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned Patent Document 1, although the walking speed is normalized according to the attributes of the resident, the attributes of the resident do not necessarily affect the walking speed in general, since elderly women are often younger than early elderly men. Therefore, the normalized walking speed may not correspond to the actual situation of the resident, and it may be difficult to detect the change in the physical condition of the resident.

[0006] In addition, the above-mentioned Cited Document 2 distinguishes whether the walking track data is that of a resident or not, and calculates the walking speed using only the data that is determined to be that of a resident. However, since the method of distinguishing is based on the starting and ending points of the track or on the time period, the walking track does not necessarily correspond to the walking track of only the resident. As a result, there is a possibility that the change in the physical condition of the resident cannot be detected properly.

[0007] The present invention has been made to solve the problems (issues) associated with the above-mentioned conventional technology, and aims to provide a monitoring system, monitoring device, monitoring method, and monitoring program that can efficiently detect changes in the physical condition of residents. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, the present invention provides a monitoring system for monitoring a resident in a room, comprising a walking trajectory data acquisition unit for acquiring walking trajectory data indicating a person's walking trajectory in the room, a walking speed calculation unit for calculating the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation unit for generating a frequency distribution by cumulatively adding the walking speeds for each speed, and a walking speed identification unit for identifying the resident's walking speed based on the frequency distribution.

[0009] In the present invention, in the above-mentioned invention, the walking speed identification unit identifies a walking speed having a highest frequency in the frequency distribution as the resident walking speed.

[0010] In the present invention, in the above-mentioned invention, the predetermined length is a predetermined measurement unit length of the walking trajectory that is determined in advance.

[0011] In the present invention, the walking trajectory data acquisition unit estimates a person's skeleton from a captured image, and acquires the walking trajectory data by acquiring chest coordinates in time series.

[0012] In addition, in the above-mentioned invention, the present invention further includes a memory unit that stores the resident walking speed, and an alarm unit that issues an alarm when the difference between the resident walking speed stored in the memory unit and the resident walking speed identified by the walking speed identification unit becomes greater than or equal to a predetermined threshold.

[0013] In the present invention, in the above-mentioned invention, the frequency distribution generating unit generates a frequency distribution for a predetermined period of time, and the storage unit stores the resident walking speed for the predetermined period of time.

[0014] In addition, in the above-mentioned invention, the present invention further includes a judgment unit that judges whether the walking trajectory data acquired by the walking trajectory data acquisition unit is suitable as a target for generating the frequency distribution, and the walking speed calculation unit calculates the walking speed when the judgment unit judges that the walking trajectory data is suitable as a target for generating the frequency distribution.

[0015] Further, in the present invention, when the walking trajectory acquired by the walking trajectory data acquisition unit is a walking trajectory of a plurality of people, the determination unit determines that the walking trajectory is not suitable as a target for generating the frequency distribution.

[0016] Further, in the present invention, in the above-mentioned invention, the determination unit determines that the walking trajectory acquired by the walking trajectory data acquisition unit is not suitable as a target for generating the frequency distribution when the walking trajectory is present within a predetermined area set in advance.

[0017] Further, in the present invention, in the above-mentioned invention, the judgment unit judges that the walking trajectory data is not suitable for generating the frequency distribution when a predetermined percentage or more of the walking trajectory data is present within a circle having a starting point and an ending point of a walking trajectory of a measurement unit length as its center and a predetermined value less than the measurement unit length as its radius.

[0018] In the present invention, in the above-mentioned invention, the predetermined length is a length of a line segment between a start point and an end point of a walking trajectory of the walking trajectory data expressed as a regression line.

[0019] The present invention also provides a monitoring device for monitoring a resident in a room, comprising a walking trajectory data acquisition unit for acquiring walking trajectory data indicating a walking trajectory of a person in the room, a walking speed calculation unit for calculating the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation unit for generating a frequency distribution by accumulating the walking speeds for each speed, and a walking speed identification unit for identifying the walking speed of the resident based on the frequency distribution.

[0020] The present invention also provides a monitoring method executed by a monitoring system for monitoring a resident in a room, the method including a walking trajectory data acquisition step of acquiring walking trajectory data indicating a walking trajectory of a person in the room, a walking speed calculation step of calculating the walking speed of the person for each predetermined length of the walking trajectory data, a frequency distribution generation step of generating a frequency distribution by accumulating the walking speeds for each speed, and a walking speed identification step of identifying the resident's walking speed based on the frequency distribution.

[0021] The present invention also provides a monitoring program executed by a monitoring device that monitors a resident in a room, which causes a computer to execute a walking trajectory data acquisition procedure for acquiring walking trajectory data indicating a walking trajectory of a person in the room, a walking speed calculation procedure for calculating the walking speed of the person for each specified length of the walking trajectory data, a frequency distribution generation procedure for generating a frequency distribution by accumulating the walking speeds for each speed, and a walking speed identification procedure for identifying the walking speed of the resident based on the frequency distribution. Effect of the Invention

[0022] According to the present invention, changes in the physical condition of a resident can be detected efficiently. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing an overview of a monitoring system according to this embodiment. [Diagram 2] FIG. 2 is a diagram showing a system configuration of the monitoring system shown in FIG. [Diagram 3]FIG. 3 is a functional block diagram showing the configuration of the imaging device shown in FIG. [Figure 4] FIG. 4 is a functional block diagram showing the configuration of the walking speed calculation device shown in FIG. [Diagram 5] FIG. 5 is a functional block diagram illustrating a configuration of the management device illustrated in FIG. [Figure 6] FIG. 6 is a functional block diagram showing a configuration of the mobile terminal shown in FIG. [Figure 7] FIG. 7 is an explanatory diagram for explaining generation of a trained model. [Figure 8] FIG. 8 is an explanatory diagram for explaining an excluded area that is not treated as walking trajectory data. [Figure 9] FIG. 9 is an explanatory diagram for explaining calculation of the chest coordinate. [Figure 10] FIG. 10 is an explanatory diagram for explaining an example of a walking trajectory and calculation of a walking speed. [Figure 11] FIG. 11 is a diagram showing an example of walking trajectory data that is not suitable for generating a frequency distribution. [Figure 12] FIG. 12 is a diagram showing an example of a frequency distribution of walking speeds. [Figure 13] FIG. 13 is a sequence diagram showing a process for identifying a walking speed of a resident in the monitoring system of FIG. [Figure 14] FIG. 14 is a flowchart showing the procedure of the image acquisition process of FIG. [Figure 15] FIG. 15 is a flowchart showing the procedure of the walking trajectory data generation process of FIG. [Figure 16] FIG. 16 is a flowchart showing the procedure of the walking frequency data suitability determination process of FIG. [Figure 17] FIG. 17 is a sequence diagram (part 1) showing the processing procedure of the monitoring system of FIG. [Figure 18] FIG. 18 is a sequence diagram (part 2) showing the processing procedure of the monitoring system of FIG. [Figure 19] FIG. 19 is a diagram showing an example of walking trajectory data of a watching system according to a modified example. [Figure 20] FIG. 20 is an explanatory diagram illustrating detection of a change point in walking direction in a monitoring system according to a modified example. [Figure 21] FIG. 21 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a monitoring system, a monitoring device, a monitoring method, and a monitoring program according to the present invention will be described in detail with reference to the accompanying drawings.

[0025] [Overview of the monitoring system] The outline of the monitoring system according to the present embodiment will be described. FIG. 1 is a diagram showing the outline of the monitoring system according to the present embodiment. As shown in FIG. 1(a), the monitoring system includes an imaging device 20, a walking speed calculation device 30, a management device 40, and a mobile terminal 50. The imaging device 20 is disposed on the ceiling of a room 90 in which a person requiring care resides, captures an image in the direction of a floor surface 91 in the room, and transmits the image to the walking speed calculation device 30. As shown in FIG. 1(b), the monitoring system according to the present embodiment obtains the walking speed of the resident based on the captured image 100, accumulates the walking speeds for each speed, and generates a frequency distribution. Then, the most frequent value of the frequency distribution is identified as the walking speed of the resident, and a change in the physical condition of the resident is detected based on the identified walking speed.

[0026] The walking speed calculation device 30 recognizes the skeleton of the resident in the room based on the image data transmitted from the imaging device 20, and acquires walking trajectory data of the resident from the chest coordinate. After that, the walking speed calculation device 30 calculates the walking speed of the resident based on the walking trajectory data. Then, the walking speed calculation device 30 transmits the walking speed of the resident to the management device 40. Note that, although details will be described later, when the walking trajectory data is in the area of ​​the bed 92 or an area close to the wall, the walking speed calculation device 30 performs a process of invalidating the walking trajectory data.

[0027] The management device 40 receives the walking speeds of the residents from the walking speed calculation device 30, and accumulates the walking speeds of the residents for each speed to generate a frequency distribution. The management device 40 then identifies the walking speed of the residents based on the frequency distribution for a predetermined period of time. Specifically, for example, a cumulative distribution of the walking speeds of the residents for each speed for one month is generated, and the most frequent walking speed in the cumulative distribution data is identified as the walking speed of the residents.

[0028] When the management device 40 receives the walking speed of the resident from the walking speed calculation device 30, it compares the received walking speed of the resident with the identified walking speed of the resident and judges whether or not the difference is equal to or greater than a predetermined threshold. If the difference is equal to or greater than the threshold, the management device 40 issues an alarm. The management device 40 also transmits walking speed judgment result data to the mobile terminal 50. The mobile terminal 50 judges whether or not to issue an alarm based on the judgment result. If the judgment result is "problematic", the mobile terminal 50 issues an alarm.

[0029] [System configuration of monitoring system 10] Next, the system configuration of the monitoring system 10 will be described. Fig. 2 is a diagram showing the system configuration of the monitoring system shown in Fig. 1. As shown in Fig. 2, the monitoring system 10 has an imaging device 20, a walking speed calculation device 30, a management device 40, and a mobile terminal 50. The walking speed calculation device 30, the management device 40, and the mobile terminal 50 are connected via a network N. In addition, the imaging device 20 is connected to the walking speed calculation device 30.

[0030] The imaging device 20 is a device that captures images of residents in the living room. The imaging device 20 includes a motor that drives an imaging unit, recognizes people based on the images captured by the imaging unit, and controls the motor so that the people are included in the angle of view of the image data.

[0031] The walking speed calculation device 30 performs a skeletal recognition process based on the image data transmitted from the imaging device 20, and acquires walking trajectory data of the resident. The walking speed calculation device 30 then determines whether the acquired walking trajectory data is suitable for calculating the walking speed, and when it is determined that the walking trajectory data is suitable for calculating the walking speed, performs a process of calculating the walking speed based on the walking trajectory data.

[0032] The management device 40 performs the following processes: generating a cumulative distribution for each speed of the walking speed data of the resident transmitted from the walking speed calculation device 30; identifying the resident's walking speed based on the cumulative distribution data of the walking speed for a specified period of time; comparing the walking speed of the resident transmitted from the walking speed calculation device 30 with the identified resident walking speed and determining whether or not the difference between the resident's walking speed and the identified resident walking speed is equal to or greater than a specified threshold; issuing an alarm if the difference is equal to or greater than the specified threshold; and transmitting to the mobile terminal 50 data on the determination result as to whether or not the difference between the resident's walking speed and the identified resident walking speed is equal to or greater than the specified threshold.

[0033] The mobile terminal 50 is a device that receives the judgment result data from the management device 40 and performs a process of displaying the status of the resident based on the judgment result data. Furthermore, the mobile terminal 50 performs a process of issuing an alarm if the judgment result data indicates "problem".

[0034] [Configuration of imaging device 20] Next, the configuration of the imaging device 20 shown in Fig. 2 will be described. Fig. 3 is a functional block diagram showing the configuration of the imaging device 20 shown in Fig. 2. As shown in Fig. 3, the imaging device 20 has an imaging unit 21, a light source unit 22, a motor 23, a communication I / F unit 24, a storage unit 25, and a control unit 26. The imaging unit 21 is a camera that captures an image of a resident in a room. Specifically, the imaging unit 21 acquires an infrared image for skeletal recognition in a walking speed calculation device 30 described later. The light source unit 22 is a device that irradiates infrared light to capture an infrared image.

[0035] The motor 23 is a drive unit used to change the angle of view of the imaging unit 21. A DC motor, a servo motor, a stepping motor, or the like is used as the motor 23, and is controlled by a motor control unit 26d described later. The communication I / F unit 24 is an interface unit for communicating with the walking speed calculation device 30, and performs wireless communication according to a mobile communication standard such as a third generation mobile communication system, a fourth generation mobile communication system, or a fifth generation mobile communication system, or short-distance wireless communication according to a wireless LAN (Local Area Network), or the like.

[0036] The storage unit 25 is a storage device such as a hard disk drive or a non-volatile memory, and stores a person recognition trained model 25a and image data 25b. The person recognition trained model 25a is a trained model downloaded from a server device (not shown). Specifically, as shown in FIG. 7, teacher data A consisting of a large number of data is input to a CNN (Convolutional Neural Network) B, and, for example, backpropagation is performed based on correct answer data R, and supervised learning for determining the weight of each path is repeated to generate a trained model M. Note that the correct answer data R, i.e., the probability that it is a known phenomenon, can be assigned by a person in charge of supervised learning. The image data 25b is image data that combines an infrared image captured by the imaging unit 21 with depth information.

[0037] The control unit 26 is a control unit that controls the entire imaging device 20, and includes a light source control unit 26a, an image acquisition unit 26b, a person recognition unit 26c, a motor control unit 26d, and an image transmission unit 26e. In practice, these programs are loaded into a CPU, a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), or the like, and executed, causing the light source control unit 26a, the image acquisition unit 26b, the person recognition unit 26c, the motor control unit 26d, and the image transmission unit 26e to execute processes corresponding thereto.

[0038] The light source control unit 26a performs on / off control processing of the light source unit 22. The image acquisition unit 26b performs processing of acquiring an infrared image using the imaging unit 21 and acquiring depth information using a ToF (Time of Flight) technique. The person recognition unit 26c is a processing unit that inputs the captured image to the person recognition trained model 25a and recognizes people.

[0039] The motor control unit 26d is a processing unit that controls the motor 23 and adjusts the angle of view of the imaging unit 21. The image transmission unit 26e is a processing unit that transmits the captured image data 25b to the walking speed calculation device 30 via the communication I / F unit 24.

[0040] [Configuration of walking speed calculation device 30] Next, the configuration of the walking speed calculation device 30 will be described. Fig. 4 is a functional block diagram showing the configuration of the walking speed calculation device 30 shown in Fig. 2. As shown in Fig. 4, the walking speed calculation device 30 has a communication I / F unit 31, a storage unit 34, and a control unit 35. The communication I / F unit 31 is an interface unit for communicating with the imaging device 20 and the management device 40, and performs wireless communication according to a mobile communication standard such as a third generation mobile communication system, a fourth generation mobile communication system, or a fifth generation mobile communication system, or short-distance wireless communication according to a wireless LAN (Local Area Network) or the like.

[0041] The storage unit 34 is a storage device such as a hard disk drive or a non-volatile memory, and stores the skeleton recognition trained model 34a, image data 34b, skeleton recognition data 34c, walking trajectory data 34d, and excluded area data 34e. The skeleton recognition trained model 34a is a trained model downloaded from a server device (not shown). The skeleton recognition trained model 34a is a trained model that inputs image data 34b and recognizes the skeleton of a person included in the image. Note that the generation of the trained model has already been described, so a detailed description will be omitted here.

[0042] The image data 34b is data of the image and depth information received from the imaging device 20. The skeleton recognition data 34c is data obtained by performing skeleton recognition on the person included in the image data 34b. The walking trajectory data 34d is data of the trajectory of the resident's walking. The excluded area data 34e is data representing an area that is not recognized as a walking trajectory among the places where the resident has walked. Specifically, as shown in FIG. 8, a bed area 94 where a bed 92 is installed and a wall area 95, etc. are set as excluded areas on a floor surface 91 of a room 90. The excluded area may be set for reasons such as the resident's normal walking speed being unable to be obtained, or the depth information being unreliable due to the performance of the camera.

[0043] The control unit 35 is a control unit that controls the entire walking speed calculation device 30, and includes an image data receiving unit 35a, a skeleton recognition unit 35b, a walking trajectory data acquisition unit 35c, a determination unit 35d, a walking speed calculation unit 35e, and a walking speed data transmission unit 35f. In practice, these programs are loaded into a CPU and executed, causing the image data receiving unit 35a, the skeleton recognition unit 35b, the walking trajectory data acquisition unit 35c, the determination unit 35d, the walking speed calculation unit 35e, and the walking speed data transmission unit 35f to execute processes corresponding to each of them.

[0044] The image data receiving unit 35a is a processing unit that receives image data transmitted from the imaging device 20 and stores the image data in the storage unit 34. The skeleton recognizing unit 35b is a processing unit that performs skeleton recognition of a person included in the image data 34b based on the image data 34b. Specifically, as shown in Fig. 9, the skeleton recognition unit 35b inputs the image 110 received from the imaging device 20 into the skeleton recognition learned model 24a to obtain a skeleton model 130 of the person 111.

[0045] In this example, the skeletal model 130 is recognized by the following parts: mouth: B0, chest: B1, right shoulder: B2, right elbow: B3, right wrist: B4, left shoulder: B5, left elbow: B6, left wrist: B7, right hip: B8, right knee: B9, right ankle: B10, left hip: B11, left knee: B12, left ankle: B13, right eye: B14, left eye: B15, right ear: B16, left eye: B17.

[0046] Thereafter, the skeleton recognition unit 35b specifies three-dimensional coordinates 140 of each part of the skeleton model 130 based on the recognized skeleton model 130 and the depth information 120.

[0047] The walking trajectory data acquisition unit 35c is a processing unit that acquires walking trajectory data 34d by acquiring chest coordinates of a person whose skeleton has been recognized in time series. The walking trajectory data acquisition unit 35c also acquires data for each predetermined measurement unit length. The predetermined measurement unit length is, for example, 2m, and when the movement distance of the time series of chest coordinates whose skeleton has been recognized reaches 2m, the data is stored in the storage unit 34 as one walking trajectory data 34d.

[0048] Specifically, as shown in FIG. 10(a), the chest coordinates of the skeleton-recognized person are acquired in time series, and the amount of change in the chest coordinates is cumulatively added. When the cumulative value reaches a predetermined measurement unit length, the walking start time is calculated as shown in FIG. 10(b). S And I finished walking. E The straight line connecting these points is stored in the memory unit 34 as walking trajectory data 34d.

[0049] The determination unit 35d is a processing unit that determines whether or not the walking speed calculated based on the acquired walking trajectory data 34d is suitable as a target for generating a frequency distribution of the management device 40 described later. Specifically, the determination unit 35d determines that the walking trajectory data 34d is inappropriate when the walking trajectories of multiple people are recorded in the walking trajectory data 34d and when the walking trajectory data 34d includes coordinates set in the excluded area data 34e.

[0050] Also, as shown in FIG. 11, the start of walking W S And I finished walking.E The walking trajectory data 34d is judged to be appropriate or not based on whether or not a certain ratio of multiple chest coordinates exists within a circle having a radius of a certain value less than a certain measurement unit length, with each of the chest coordinates being the center. For example, if the certain ratio is set to 60%, as shown in FIG. 11(a), S And I finished walking. E There are multiple chest coordinates within a circle of a specified radius with each of the points at the center, but the ratio is less than 60%, so this walking trajectory data 34d is determined to be appropriate. S And I finished walking. E If 60% or more of the chest coordinates are within a circle of a predetermined radius centered on each of the points, the walking trajectory data 34d is determined to be inappropriate.

[0051] The walking speed calculation unit 35e is a processing unit that calculates the walking speed based on the walking trajectory data 34d and the time required to walk a predetermined measurement unit length. The walking speed data transmission unit 35f is a processing unit that transmits the calculated walking speed data to the management device 40.

[0052] [Configuration of management device 40] Next, the configuration of the management device 40 will be described. Fig. 5 is a functional block diagram showing the configuration of the management device 40 shown in Fig. 2. As shown in Fig. 5, the management device 40 has a display unit 41, an input unit 42, a communication I / F unit 43, a storage unit 44, and a control unit 45. The display unit 41 is a display device such as a liquid crystal display that displays various information.

[0053] The input unit 42 is an input device such as a mouse or a keyboard. The communication I / F unit 43 is an interface unit for communicating with the walking speed calculation device 30 and the mobile terminal 50, and performs wireless communication according to a mobile communication standard such as a third generation mobile communication system, a fourth generation mobile communication system, or a fifth generation mobile communication system, or short-distance wireless communication according to a wireless LAN (Local Area Network) or the like.

[0054] The storage unit 34 is a storage device such as a hard disk drive or a non-volatile memory, and stores walking speed data 44a, walking speed frequency distribution data 44b, specific resident walking speed data 44c, and judgment result data 44d. The walking speed data 44a is data on the walking speed of the resident received from the walking speed calculation device 30. The walking speed frequency distribution data 44b is data obtained by accumulating the walking speed for each speed.

[0055] The specific resident walking speed data 44c is data that identifies the most frequent walking speed based on the walking speed frequency distribution data 44b for a predetermined period. The judgment result data 44d is data of the judgment result that judges whether the difference between the resident walking speed data 44a and the specific resident walking speed data 44c of the resident is within a predetermined threshold value or not.

[0056] The control unit 45 is a control unit that controls the entire management device 40, and includes a walking speed data receiving unit 45a, a frequency distribution generating unit 45b, a walking speed specifying unit 45c, a walking speed determining unit 45d, a notifying unit 45e, and a determination result data transmitting unit 45f. In practice, by loading these programs into a CPU and executing them, the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the determination result data transmitting unit 45f execute processes corresponding to them, respectively.

[0057] The walking speed data receiving unit 45a is a processing unit that receives the walking speed data of the resident calculated by the walking speed calculation device 30 and stores it as walking speed data 44a in the storage unit 44. The frequency distribution generating unit 45b is a processing unit that accumulates the walking speed data 44a of the resident for each walking speed.

[0058] The walking speed identification unit 45c is a processing unit that identifies the walking speed of the resident in a predetermined period based on the walking speed frequency distribution data 44b, and stores it as specific resident walking speed data 44c in the storage unit 44. For example, as shown in Fig. 12, the walking speed is classified in a range from 0 to 1.0 m / s in increments of 0.05 m / s, and the most frequent walking speed of 0.25 m / s to 0.30 m / s is identified as the walking speed of the resident in the predetermined period based on the cumulative distribution data for each speed.

[0059] The walking speed determination unit 45d is a processing unit that compares the walking speed data 44a transmitted from the walking speed calculation device 30 with the specific resident walking speed data 44c for the specified period specified by the walking speed specification unit 45c, and determines whether the difference is equal to or greater than a specified threshold. If the difference between the walking speed data 44a transmitted from the walking speed calculation device 30 and the specific resident walking speed data 44c for the specified period specified by the walking speed specification unit 45c is within the specified threshold, the walking speed determination unit 45d determines that the walking speed is "no problem", and if the difference is greater than the specified threshold, the walking speed determination unit 45d determines that the walking speed is "problematic". After that, the walking speed determination unit 45d stores the determination result data 44d in the storage unit 44.

[0060] The notification unit 45e is a processing unit that performs notification based on the judgment result data 44d. Specifically, when the judgment result data 44d is "problem", the notification unit 45e displays the walking speed on the predetermined display unit 41 and generates an alarm sound. When the judgment result data 44d is "no problem", the notification unit 45e displays the walking speed on the predetermined display unit 41.

[0061] The determination result data transmission unit 45f is a processing unit that transmits the determination result data 44d to the mobile terminal 50.

[0062] [Configuration of mobile terminal 50] Next, the configuration of the mobile terminal 50 will be described. Fig. 6 is a functional block diagram showing the configuration of the mobile terminal 50 shown in Fig. 2. As shown in Fig. 6, the mobile terminal 50 has an operation display unit 51, a communication I / F unit 52, a storage unit 54, and a control unit 55. The operation display unit 51 is an input and display device such as a touch panel type liquid crystal display.

[0063] The communication I / F unit 52 is an interface unit that communicates with the management device 40, and performs wireless communication according to a mobile communication standard such as a third generation mobile communication system, a fourth generation mobile communication system, or a fifth generation mobile communication system, or short-distance wireless communication according to a wireless LAN (Local Area Network), etc. The storage unit 54 is a storage device such as a hard disk drive or a nonvolatile memory.

[0064] The control unit 55 is a control unit that controls the entire mobile terminal 50, and has a determination result data receiving unit 55a and a notification unit 55b. In practice, by loading these programs into the CPU and executing them, the determination result data receiving unit 55a and the notification unit 55b are caused to execute processes corresponding to them, respectively.

[0065] The judgment result data receiving unit 55a is a processing unit that receives the judgment result data 44d transmitted from the management device 40. The notification unit 55b is a processing unit that issues a notification based on the judgment result data 44d. Specifically, when the judgment result data 44d is "problematic", the notification unit 55b displays the walking speed on the predetermined operation display unit 51 and issues an alarm sound. When the judgment result data 44d is "no problem", the notification unit 55b displays the walking speed on the predetermined operation display unit 51.

[0066] [Processing procedure for identifying resident walking speed in monitoring system] Next, a process procedure for identifying a resident's walking speed in the monitoring system 10 will be described. Fig. 13 is a sequence diagram showing a process procedure for identifying a resident's walking speed in the monitoring system 10 of Fig. 2. As shown in Fig. 13, the imaging device 20 of the monitoring system 10 performs an image acquisition process (step S101). Then, the imaging device 20 transmits image data to the walking speed calculation device 30 (step S102).

[0067] The walking speed calculation device 30 determines whether or not image data has been received from the imaging device 20 (Step S103). If image data has not been received, the walking speed calculation device 30 waits until image data is received (Step S103: No).

[0068] When the walking speed calculation device 30 receives the image data (step S103: Yes), it performs a walking trajectory data generation process (step S104). Then, the walking speed calculation device 30 judges whether the movement distance of the resident exceeds a predetermined threshold (measurement unit length) or not (step S105). When the movement distance does not exceed the predetermined threshold (step S105: No), the walking speed calculation device 30 proceeds to step S103 and waits for the reception of image data from the imaging device 20.

[0069] On the other hand, when the moving distance exceeds a predetermined threshold (step S105: Yes), the walking speed calculation device 30 performs a walking trajectory data appropriateness determination process (step S106). Then, the walking speed calculation device 30 determines whether the walking trajectory data is appropriate as data for calculating the walking speed and accumulating the frequency distribution data (step S107). When the walking trajectory data is not appropriate (step S107: No), the walking speed calculation device 30 proceeds to step S103 and waits for image data to be transmitted from the imaging device 20.

[0070] On the other hand, if the walking trajectory data is appropriate (step S107: Yes), the walking speed calculation device 30 calculates the walking speed (step S108). After that, the walking speed calculation device 30 transmits the walking speed data to the management device 40 (step S109).

[0071] The management device 40 judges whether or not walking speed data has been received from the walking speed calculation device 30 (step S110). If the management device 40 has not received walking speed data (step S110: No), it waits for reception of walking speed data. On the other hand, if the management device 40 receives walking speed data from the walking speed calculation device 30 (step S110: Yes), it cumulatively adds the received walking speed data to the walking speed frequency distribution data for each speed (step S111).

[0072] Then, the management device 40 judges whether or not a predetermined period of time has elapsed (step S112). If the predetermined period of time has not elapsed (step S112: No), the management device 40 ends the process. On the other hand, if the predetermined period of time has elapsed (step S112: Yes), the management device 40 identifies the most frequently occurring walking speed as the resident walking speed based on the walking speed frequency distribution data (step S113).

[0073] [Image acquisition processing procedure] Next, a process procedure of the image acquisition process shown in Fig. 13 will be described. Fig. 14 is a flowchart showing the process procedure of the image acquisition process in Fig. 13. The imaging device 20 captures an image of the living room 90 using the imaging unit 21 (step S201). Then, the imaging device 20 recognizes a person from the captured image data (step S202).

[0074] After that, the imaging device 20 adjusts the angle of the imaging unit so that the person is within the imaging range (step S203). Then, the imaging device 20 captures an image of the living room 90 using the imaging unit 21 (step S204), and the process returns to step S102 in FIG.

[0075] [Processing procedure for generating walking trajectory data] Next, a description will be given of the process of generating walking trajectory data shown in Fig. 13. Fig. 15 is a flowchart showing the process of generating walking trajectory data shown in Fig. 13. As shown in Fig. 15, the walking speed calculation device 30 performs skeleton recognition based on image data (step S301).

[0076] Then, the walking speed calculation device 30 acquires chest coordinates based on the skeleton recognition data (step S302). After that, the walking speed calculation device 30 stores the chest coordinates as a walking trajectory (step S303). After that, the walking speed calculation device 30 calculates the moving distance from the difference in the chest coordinates (step S304) and returns to step S105 in FIG.

[0077] [Processing procedure for determining whether walking frequency data is appropriate] Next, the process procedure of the walking frequency data suitability judgment process of Fig. 13 will be described. Fig. 16 is a flowchart showing the process procedure of the walking frequency data suitability judgment process of Fig. 13. As shown in Fig. 16, the walking speed calculation device 30 judges whether the walking trajectory data 34d is data of one or more people (step S401). If no one is captured (step S401: No), the walking speed calculation device 30 judges that the walking trajectory data 34d is "inappropriate" (step S407) and returns to S107 of Fig. 13.

[0078] On the other hand, if the number of people shown is one or more (step S401: Yes), the walking speed calculation device 30 judges whether there is one person moving (step S402). If there are two or more people moving (step S402: No), the walking speed calculation device 30 judges that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG. 13.

[0079] On the other hand, if there is only one moving person (step S402: Yes), the walking speed calculation device 30 judges whether the chest coordinate is outside the exclusion area (step S403). If the chest coordinate is in the exclusion area (step S403: No), the walking trajectory data 34d is judged to be "inappropriate" (step S407), and the process returns to S107 in FIG. 13.

[0080] On the other hand, if the chest coordinate is outside the exclusion area (step S403: Yes), the walking speed calculation device 30 judges whether the up-down movement of the chest coordinate is equal to or smaller than a predetermined threshold (step S404). If the up-down movement of the chest coordinate is greater than the predetermined threshold (step S404: No), the walking trajectory data 34d is judged to be "inappropriate" (step S407), and the process returns to S107 in FIG. 13.

[0081] On the other hand, if the up-down movement of the chest coordinate is equal to or smaller than the predetermined threshold (step S404: Yes), the walking speed calculation device 30 judges whether the chest coordinate is present in a circle of a predetermined size having the start and end of the walk as a predetermined percentage or more (step S405).If the chest coordinate is present in a circle of a predetermined size having the start and end of the walk as a center, the walking speed calculation device 30 judges that the walking trajectory data 34d is "inappropriate" (step S407), and returns to S107 in FIG.

[0082] On the other hand, if the chest coordinate does not exist at a rate equal to or greater than the predetermined rate (step S405: Yes), the walking speed calculation device 30 determines that the walking trajectory data 34d is "appropriate" (step S406), and returns to step S107 in FIG.

[0083] [Processing procedure of monitoring system 10] Next, a processing procedure for monitoring a resident by the monitoring system 10 shown in Fig. 2 will be described. Figs. 17 and 18 are sequence diagrams showing the processing procedure of the monitoring system 10 of Fig. 2. As shown in Fig. 17, the imaging device 20 of the monitoring system 10 performs an image acquisition process (step S501). Then, the imaging device 20 transmits image data to the walking speed calculation device 30 (step S502). Note that the image acquisition process has already been described, so a detailed description will be omitted here.

[0084] The walking speed calculation device 30 determines whether or not image data has been received from the imaging device 20 (Step S503). If image data has not been received, the walking speed calculation device 30 waits until image data is received (Step S503: No).

[0085] When the walking speed calculation device 30 receives the image data (step S503: Yes), it performs a walking trajectory data generation process (step S504). Then, the walking speed calculation device 30 judges whether or not the movement distance of the resident exceeds a predetermined threshold (step S505). When the movement distance does not exceed the predetermined threshold (step S505: No), the walking speed calculation device 30 proceeds to step S503 and waits for reception of image data from the imaging device 20. Note that the walking trajectory data generation process has already been described, so a detailed description will be omitted here.

[0086] On the other hand, when the moving distance exceeds the predetermined threshold (step S505: Yes), the walking speed calculation device 30 performs a walking trajectory data suitability determination process (step S506). Then, the walking speed calculation device 30 determines whether the walking trajectory data is appropriate as data for calculating the walking speed and accumulating the frequency distribution data (step S507). When the walking trajectory data is not appropriate (step S507: No), the walking speed calculation device 30 proceeds to step S503 and waits for image data to be transmitted from the imaging device 20. Note that the walking trajectory data suitability determination process has already been described, so a detailed description thereof will be omitted here.

[0087] On the other hand, if the walking trajectory data is appropriate (step S507: Yes), the walking speed calculation device 30 calculates the walking speed (step S508). After that, the walking speed calculation device 30 transmits the walking speed data to the management device 40 (step S509).

[0088] The management device 40 judges whether or not walking speed data has been received from the walking speed calculation device 30 (step S510). If the management device 40 has not received walking speed data (step S510: No), it waits for reception of walking speed data. On the other hand, if the management device 40 receives walking speed data from the walking speed calculation device 30 (step S510: Yes), it reads out the specific resident walking speed data 44c identified by the walking speed identification unit 45c (step S511) and compares the walking speed data with the specific resident walking speed data 44c (step S512).

[0089] If the difference between the walking speed and the specific resident's walking speed is within a predetermined threshold (step S513: Yes), the management device 40 determines that the walking speed is "not problematic" (step S514). On the other hand, if the difference between the walking speed and the specific resident's walking speed is not within the predetermined threshold (step S513: No), the management device 40 determines that the walking speed is "problematic" (step S515).

[0090] Thereafter, the management device 40 transmits the judgment result to the mobile terminal 50 (step S516), and displays the judgment result data on a predetermined display unit 41 (step S517). The management device 40 then judges whether the judgment result is "problematic" or not (step S518). If the judgment result is "problematic" (step S518: Yes), a warning is issued (step S519). The warning is issued by displaying a message such as "The walking speed of resident XX has decreased by 20%." On the other hand, if the judgment result is "not problematic" (step S518: No), the management device 40 ends the process.

[0091] When the mobile terminal 50 receives the judgment result data (step S520), it displays the received judgment result data on a predetermined operation display unit 51 (step S521). Then, the mobile terminal 50 judges whether or not the judgment result is "problematic" (step S522). If the judgment result is "problematic" (step S522: Yes), it issues a warning (step S523). The warning is issued by displaying a message such as "The walking speed of resident XX has decreased by 20%." On the other hand, if the judgment result is "not problematic" (step S522: No), the mobile terminal 50 ends the process.

[0092] As described above, in this embodiment, the monitoring system 10 captures an image of the resident using the imaging device 20, and the walking speed calculation device 30 performs skeletal recognition based on the image and acquires chest coordinates in a time series to acquire walking trajectory data. After that, the walking speed calculation device 30 calculates the walking speed based on the walking trajectory data. The management device 40 accumulates the calculated walking speeds for each speed to generate a frequency distribution, and identifies the walking speed of the resident for a predetermined period based on the frequency distribution. After that, the management device 40 calculates the difference between the acquired walking speed and the identified walking speed, and determines whether the difference is equal to or greater than a predetermined threshold. If the difference is equal to or greater than the predetermined threshold, a notification is issued. In addition, the mobile terminal 50 also receives the result of the determination, and if the difference is equal to or greater than a predetermined threshold, a notification is issued, so that changes in the physical condition of the resident can be detected efficiently.

[0093] <Modification> In the above embodiment, the monitoring system 10 detects the walking path of the resident from the start of walking W S And I finished walking. E In the above, the case where the walking trajectory of the resident is represented by a single regression line has been described, but in the modified example, a case where the walking trajectory of the resident is not represented by a single regression line will be described.

[0094] 19A and 19B are diagrams showing an example of walking track data of a monitoring system according to a modified example. As shown in FIG. 19A, when a walking track of a resident draws multiple curves, a predetermined measurement unit length (for example, W S From W E Even if the resident starts walking, the distance to the S The first change point W where the walking direction of the resident's walking trajectory changes from n Alternatively, a regression line may be calculated to the distance, and the walking speed may be calculated based on the time required to move this distance and the length L1 of the regression line.

[0095] Furthermore, as shown in FIG. 19(b), when the resident further changes the walking direction, the first change point W n Starting from the second change point W m Alternatively, a regression line may be calculated to calculate the walking speed based on the time required to move this distance and the length L2 of the regression line.

[0096] Furthermore, as shown in FIG. 19(c), the monitoring system detects the second change point W m After walking from E Alternatively, the walking speed may be calculated based on the time required to move this distance and the length L3 of the regression line.

[0097] [About change point detection] Next, detection of a change in the walking direction of a resident by the monitoring system according to the modified example will be described. FIG. 20 is an explanatory diagram for explaining detection of a change in the walking direction by the monitoring system according to the modified example. As shown in FIG. 20(a), when a resident starts walking, S With the origin at g0, the coordinates of the time series data of the chest coordinates are g0, g1, and g2, respectively.

[0098] Then, as shown in Figure 20(b), the walking start W s Starting from the starting point, start walking W s The angle on the XY plane between the time series data g0 of the first chest coordinate and the position of the first chest coordinate is set as the reference angle. sThe difference θ1 between the angle on the XY plane formed by the time series data g1 of the second chest coordinate and the reference angle is calculated. s A difference θ2 between the angle on the XY plane formed by the time series data g2 of the third chest coordinate and the reference angle is calculated.

[0099] The monitoring system starts walking like this. s is set as the origin, and the angle on the XY plane formed by the first chest coordinate time series data g0 is set as the reference angle. If the difference between the coordinate of each chest coordinate time series data and the reference angle exceeds a predetermined threshold, it is determined that the resident has changed his / her walking direction.

[0100] In the above embodiment, the management device 40 of the monitoring system 10 is described as transmitting the judgment result data of the resident's walking speed to the mobile terminal 50. However, the management device 40 may be configured to transmit warning data to the mobile terminal 50 when the judgment result data indicates "problem."

[0101] <Relationship with hardware> Next, the correspondence between the management device 40 of the watching system 10 according to this embodiment and the main hardware configuration of a computer will be described. Fig. 21 is a diagram showing an example of the hardware configuration.

[0102] Generally, a computer has a configuration in which a CPU 81, a ROM 82, a RAM 83, a non-volatile memory 84, etc. are connected via a bus 85. A hard disk device may be provided instead of the non-volatile memory 84. For the sake of convenience of explanation, only the basic hardware configuration is shown.

[0103] Here, the ROM 82 or the non-volatile memory 84 stores programs and the like required to start an operating system (hereinafter simply referred to as "OS"), and when the power is turned on, the CPU 81 reads and executes the OS program from the ROM 82 or the non-volatile memory 84.

[0104] Meanwhile, various application programs executed on the OS are stored in non-volatile memory 84, and the CPU 81 executes the application programs while using the RAM 83 as the main memory, thereby executing processes corresponding to the applications.

[0105] The monitoring program of the management device 40 of the monitoring system 10 according to this embodiment is stored in the non-volatile memory 84 or the like, like other application programs, and the CPU 81 loads and executes the treatment management program. In the case of the management device 40 of the monitoring system 10 according to this embodiment, a monitoring program including routines corresponding to the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the judgment result data transmitting unit 45f shown in FIG. 5 is stored in the non-volatile memory 84 or the like. The CPU 81 loads and executes the monitoring program, thereby generating a monitoring process corresponding to the walking speed data receiving unit 45a, the frequency distribution generating unit 45b, the walking speed specifying unit 45c, the walking speed determining unit 45d, the notifying unit 45e, and the judgment result data transmitting unit 45f.

[0106] The configurations illustrated in the above embodiments are merely functional schematics, and are not necessarily physically configured as illustrated. In other words, the form of distribution and integration of each device is not limited to that illustrated, and all or part of the devices can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. [Industrial Applicability]

[0107] The monitoring system, monitoring device, monitoring method, and monitoring program according to the present invention are suitable for efficiently detecting changes in the physical condition of a resident. [Explanation of symbols]

[0108] 10. Monitoring System 20 Imaging device 21 Imaging unit 22 Light source section 23 Motor 24 Communication I / F section 25 Memory section 25a Person recognition trained model 25b Image data 26 Control Unit 26a Light source control unit 26b Image acquisition section 26c Person recognition section 26d Motor control unit 27e Image transmission unit 30 Walking speed calculation device 31 Communication I / F section 34 Storage section 34a Skeleton recognition trained model 34b Image data 34c Skeleton recognition data 34d Walking track data 34e Exclusion Area Data 35 Control section 35a Image data receiving unit 35b Skeleton Recognition Unit 35c Walking track data acquisition section 35d Judgment section 35e Walking speed calculation unit 35f Walking speed data transmission unit 40 Management device 41 Display section 42 Input section 43 Communication I / F section 44 Memory section 44a Walking speed data 44b Walking speed frequency distribution data 44c Specific resident walking speed data 44d Judgment result data 45 Control section 45a Walking speed data receiver 45b Frequency distribution generation unit 45c Walking speed determination section 45d Walking speed determination unit 45e Notification Department 45f Judgment result data transmission unit 50 Mobile Devices 51 Operation display section 52 Communication I / F section 54 Memory section 55 Control section 55a Judgment result data receiving unit 55b Notification Department 81 CPU 82 ROM 83 RAM 84 Non-volatile memory 85 Bus 90 Room 91 Floor 92 Beds 93 Door 94 Bed Area 95 Wall area 100 images 110 Infrared Imaging 111 People 120 Depth information 130 Skeleton Model 140 3D coordinates A. Teacher data BCNN M trained models N Network R correct answer data

Claims

1. A monitoring system for monitoring a resident in a room, comprising: a walking trajectory data acquisition unit that acquires walking trajectory data indicating a walking trajectory of a person in the living room; a walking speed calculation unit that calculates a walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generating unit that accumulates the walking speeds for each speed to generate a frequency distribution; a walking speed identification unit that identifies a walking speed of the resident based on the frequency distribution; A monitoring system comprising:

2. The walking speed identification unit is The monitoring system according to claim 1 , wherein the walking speed having the highest frequency in the frequency distribution is identified as the resident's walking speed.

3. The predetermined length is The monitoring system according to claim 1 , wherein the unit length is a predetermined measurement unit length of the walking trajectory.

4. The walking trajectory data acquisition unit is The monitoring system according to claim 1, wherein the walking trajectory data is acquired by estimating the person's skeleton from a captured image and acquiring the chest coordinates in time series.

5. A storage unit that stores the resident walking speed; a notification unit that notifies when a difference between the resident walking speed stored in the storage unit and the resident walking speed identified by the walking speed identification unit becomes equal to or greater than a predetermined threshold; The monitoring system according to any one of claims 1 to 4, further comprising:

6. the frequency distribution generating unit generates the frequency distribution for a predetermined period of time; The storage unit stores the resident walking speed during the predetermined period. The monitoring system according to claim 5 .

7. A determination unit that determines whether the walking trajectory data acquired by the walking trajectory data acquisition unit is suitable as a target for generating the frequency distribution, The walking speed calculation unit The monitoring system according to claim 1 , wherein the walking speed is calculated when the determining unit determines that the person is suitable as a subject for generating the frequency distribution.

8. The determination unit is The monitoring system according to claim 7, characterized in that when the walking trajectory acquired by the walking trajectory data acquisition unit is the walking trajectory of a plurality of people, it is determined that the walking trajectory is not suitable as a target for generating the frequency distribution.

9. The determination unit is The monitoring system according to claim 7, characterized in that, when the walking trajectory acquired by the walking trajectory data acquisition unit exists within a predetermined specified area, it is determined that the walking trajectory is not suitable for use as a target for generating the frequency distribution.

10. The determination unit is The monitoring system described in claim 7, characterized in that if a predetermined percentage or more of the walking trajectory data is present within a circle whose center is the start point and end point of a walking trajectory of a measurement unit length and whose radius is a predetermined value less than the measurement unit length, it is determined that the walking trajectory is not suitable for generating the frequency distribution.

11. The predetermined length is The monitoring system according to claim 1, wherein the walking trajectory of the walking trajectory data is a line segment length between a start point and an end point expressed as a regression line.

12. A monitoring device for monitoring a resident in a room, a walking trajectory data acquisition unit that acquires walking trajectory data indicating a walking trajectory of a person in the living room; a walking speed calculation unit that calculates a walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generating unit that accumulates the walking speeds for each speed to generate a frequency distribution; a walking speed identification unit that identifies a walking speed of the resident based on the frequency distribution; A monitoring device comprising:

13. A monitoring method executed in a monitoring system for monitoring a resident in a room, comprising: a walking trajectory data acquisition step of acquiring walking trajectory data indicating a walking trajectory of a person in the living room; a walking speed calculation step of calculating a walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generating step of generating a frequency distribution by cumulatively adding the walking speeds for each speed; a walking speed identification step of identifying a walking speed of the resident based on the frequency distribution; A monitoring method comprising:

14. A monitoring program executed by a monitoring device that monitors a resident in a room, A walking trajectory data acquisition step of acquiring walking trajectory data indicating a walking trajectory of a person in the living room; a walking speed calculation step of calculating a walking speed of the person for each predetermined length of the walking trajectory data; a frequency distribution generating step of generating a frequency distribution by cumulatively adding the walking speeds for each speed; a walking speed identification step of identifying a walking speed of the resident based on the frequency distribution; A monitoring program characterized by causing a computer to execute the above.

Images