Care recording system, care recording method, and care recording program

Abstract

To provide a care recording system, a care recording method, and a care recording program which can specify whose action where judged.SOLUTION: A care recording system 1 has an acquisition part 211, an authentication part 213, a determination part 214, and an output part 215. The acquisition part 211 acquires an image including a person. The authentication part 213 performs face authentication of the person, and specifies the person. The determination part 214 determines the action of the person, on the basis of an image. The output part 215 outputs the information on the action of the person, on the basis of the authentication result by the authentication part 213, and the determination result by the determination part 214.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a care record system, a care record method, and a care record program.

Background Art

[0002] In Japan, due to the improvement of living standards, the improvement of sanitary environment, and the improvement of medical standards accompanying the high economic growth after the war, the aging of the population has become remarkable. Therefore, combined with the decline in the birth rate, it has become an aging society with a high aging rate. In such an aging society, an increase in the number of care recipients and nursing recipients (hereinafter referred to as "care target persons") who require care, nursing, etc. due to illness, injury, aging, etc. is assumed. In care facilities such as hospitals and elderly welfare facilities (hereinafter also simply referred to as "facilities"), care for care target persons is provided by care staff such as nurses and caregivers (hereinafter referred to as "care staff").

[0003] Care staff are required to record the care status of care target persons in care records every day in order to grasp the health status and daily life status of care target persons in the facility and to assist in creating a care plan. Recording in care records is important for providing sufficient care to care target persons, but with the increase in the number of care target persons, the increase in the burden on care staff has become an issue.

[0004] In this regard, Patent Document 1 below discloses a care record creation system aimed at suppressing an increase in the burden on caregivers related to entry into care records or medical records. This care record creation system senses the behavior or state of a care recipient by a sensor, converts the sensing information obtained from the sensor into information used in care records, and records the information converted by the conversion means as character information in care records.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the care record creation system described in Patent Document 1 above, it is impossible to specify whether the action detected by the sensor is the action of the care recipient or the action of another person (for example, a caregiver). For example, in this system, "when the voice obtained from the microphone sensor is similar to the feature amount A6, or when the video obtained from the camera sensor is similar to the feature amount V6, 'fell down' is described in the care record" (paragraph

[0063] ). With such a configuration, at the location where the sensor is installed, it is detected that someone "fell down", but it is impossible to specify whether the actual person who fell is the care recipient or another person (for example, a caregiver). Therefore, the description regarding the fall in the care record created by the care record creation system may be inaccurate.

[0007] The present invention has been made in view of the above circumstances, and the main object is to provide a care record system, a care record method, and a care record program that can specify who the determined action belongs to.

Means for Solving the Problems

[0008] The above problems of the present invention are solved by the following means.

[0009] (1) A care record system comprising: an acquisition unit that acquires an image including a person; an authentication unit that performs face authentication of the person to identify the person; a determination unit that determines the action of the person based on the image; and an output unit that outputs information regarding the action of the person based on the authentication result by the authentication unit and the determination result by the determination unit.

[0010] (2) The care record system according to (1) above further includes an estimation unit that estimates the joint points of the person based on the image, and the determination unit determines the behavior of the person based on the joint points estimated by the estimation unit or the temporal change in the position of the skeleton connecting the joint points.

[0011] (3) The authentication unit authenticates the person based on the image, and the care record system according to (1) or (2) above.

[0012] (4) The authentication unit authenticates that a plurality of persons included in the image are pre-registered caregivers and care recipients, and the care record system according to (3) above.

[0013] (5) The determination unit specifies the content of the care provided by the caregiver to the care recipient based on the determination result and outputs it as information regarding the behavior of the person, and the care record system according to (4) above.

[0014] (6) The output unit outputs the information regarding the behavior of the person to the terminal of the caregiver as a care record plan for the care recipient, and the care record system according to (5) above.

[0015] (7) The care record system according to (6) above further includes a reception unit that receives changes to the care record plan.

[0016] (8) The care record system according to (4) above further includes a storage unit that stores the care record of the care recipient, and the output unit updates the content of the care record using the information regarding the behavior of the person.

[0017] (9) The care record system according to (1) or (2) above further includes a photographing unit that photographs the person, and the determination unit acquires information regarding the position where the photographing unit photographs the person and extracts candidates for the behavior of the person based on the information.

[0018] (10) The output unit outputs information regarding the action when the person starts a specific action, for the care recording system according to (1) or (2) above.

[0019] (11) The determination unit determines the action of the person based on the image and the authentication result, for the care recording system according to (1) or (2) above.

[0020] (12) A care recording method including: a step (a) of acquiring an image including a person; a step (b) of performing face authentication of the person to identify the person; a step (c) of determining the action of the person based on the image; and a step (d) of outputting information regarding the action of the person based on the authentication result in step (b) and the determination result in step (c).

[0021] (13) A care recording program for causing a computer to execute a process including: a procedure (a) of acquiring an image including a person; a procedure (b) of performing face authentication of the person to identify the person; a procedure (c) of determining the action of the person based on the image; and a procedure (d) of outputting information regarding the action of the person based on the authentication result in procedure (b) and the determination result in procedure (c).

Advantages of the Invention

[0022] According to the present invention, it is possible to identify whose action the determined action is.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0024] Hereinafter, with reference to the drawings, a care record system, a care record method, and a care record program according to embodiments of the present invention will be described. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for the sake of explanation and may be different from the actual ratios.

[0025] (Embodiment) <Care Record System 1> FIG. 1 is a diagram illustrating the overall configuration of a care record system 1 according to an embodiment. The care record system 1 includes a plurality of detection units 10, a server 20, a fixed terminal 30, one or more portable terminals 40, and cameras 51 to 54 arranged in each shared space or near the shared space. The shared space can be, for example, a cafeteria, a washroom, a toilet, a bathroom, a lobby, a corridor, etc. The cameras 51 to 54 function as photographing units and are arranged mainly for the purpose of photographing people (for example, care recipients 80, care staff 70) in the shared space. On the other hand, from the viewpoint of protecting the privacy of people in the shared space, for toilets and bath areas, for example, cameras 53 and 54 can be arranged near the entrances and exits (for example, in front of the toilet or in front of the bathroom).

[0026] In the present embodiment, the cameras 51 to 54 may have a lidar (Light Detection and Ranging). The lidar irradiates a pulse of laser light onto the target object, detects the reflected light, and measures the distance to the target object. The lidar generates a 3D map (distance image) of the body using the measured distance values as pixel values and transmits the distance image to the control unit 21.

[0027] The detection unit 10, the server 20, the fixed terminal 30, the mobile terminal 40, and the cameras 51 to 54 are communicably connected to each other via a network 60 such as a LAN (Local Area Network), a telephone network, or a data communication network, either by wire or wirelessly. The network 60 may include a relay device that relays communication signals. In the example shown in FIG. 1, the detection unit 10, the server 20, the fixed terminal 30, and the mobile terminal 40 are communicably connected to each other by a network 60 such as a wireless LAN (e.g., a LAN conforming to the IEEE802.11 standard) including an access point 61. The care record system 1 constitutes a care workload calculation system. The care workload calculation system may be constituted by only the server 20.

[0028] The care record system 1 is preferably disposed, for example, in a building such as a hospital or a nursing home. In the example shown in FIG. 1, the care record system 1 is disposed inside a building of a facility including a plurality of rooms including a plurality of living rooms or care stations in which a plurality of care recipients (care receivers) 80 each reside. Hereinafter, the living rooms in which the care recipients 80 each reside are also simply referred to as "living rooms".

[0029] The detection unit 10 is disposed in each living room on each floor, which is an observation area of the care recipient 80. In the example shown in FIG. 1, two detection units 10 are respectively disposed in the living rooms of Mr. A and Mr. B, who are care recipients 80. Although not shown, in other living rooms, a plurality of detection units 10 are respectively disposed in the living rooms of the care recipients 80. The care staff (caregivers) 70 each carry a mobile terminal 40. The server 20 may be an external server connected to the network 60. Also, the fixed terminal 30 may be omitted, and the server 20 or the mobile terminal 40 may have its function.

[0030] <Detection unit 10> FIG. 2 is a diagram showing the interior of the living room of the care recipient 80 in which the detection unit 10 shown in FIG. 1 is disposed. FIG. 3 is a block diagram illustrating the hardware configuration of the detection unit 10 shown in FIG. 1.

[0031] As shown in FIG. 2, the camera 14 and the body motion sensor 15 are arranged, for example, on the ceiling of a living room. The camera 14 and the body motion sensor 15 may be arranged at the upper part of a wall or attached to the bed 90. The care call unit 16 is arranged, for example, at an appropriate position on the wall.

[0032] As shown in FIG. 3, the detection unit 10 includes a control unit 11, a communication unit 12, a storage unit 13, a camera 14, a body motion sensor 15, and a care call unit 16, and these components are interconnected by a bus. Each component may be mounted in a single housing or in individual housings respectively.

[0033] The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The control unit 11 performs control and arithmetic processing of each part of the detection unit 10 according to a program.

[0034] The communication unit 12 is an interface circuit (such as a LAN card, a wireless communication circuit, etc.) for communicating with, for example, a server 20 via a LAN.

[0035] The storage unit 13 includes an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The storage unit 13 stores various programs and various data.

[0036] The camera 14 can image a shooting area including, for example, the bed 90 and output a captured image (image data). The camera 14 may be configured to include an imaging optical system and a two-dimensional image sensor. The captured image includes still images and moving images. The camera 14 may be, for example, a visible light camera or a near-infrared camera. Hereinafter, the captured image captured by the camera 14 is also simply referred to as "captured image".

[0037] The body movement sensor 15 transmits microwaves to a predetermined irradiation area and receives the reflected waves, thereby detecting the Doppler shift of the microwaves caused by the body movement (e.g., respiratory movement) of the care recipient 80 as the body movement. The predetermined irradiation area is an area where the care recipient 80 can be irradiated with microwaves, and can be, for example, an area including part or all of the bed 90. The body movement sensor 15 detects the body movement of the chest (the up and down movement of the chest) accompanying the respiratory movement of the care recipient 80, and detects the disturbance of the period in the body movement of the chest and the amplitude in the body movement of the chest below a preset threshold value, thereby detecting a minute body movement abnormality.

[0038] The care call unit 16 includes a push-button type switch, and detects a care call when the switch is pushed by the care recipient 80. Instead of the push-button type switch, a care call may be detected based on the voice of the care recipient 80 by a voice microphone. Note that the care call unit 16 may include a microphone and a speaker for voice calls with the mobile terminal 40.

[0039] The control unit 11 recognizes the actions of the care recipient 80 from the captured images. The actions to be recognized include, for example, getting up, getting out of bed, getting out of a seat, falling, tumbling, walking, and going out.

[0040] The control unit 11 detects the silhouette of an image (for example, a moving image) from a plurality of captured images (hereinafter referred to as "human silhouette"). The human silhouette can be detected, for example, by extracting a range of pixels with a relatively large difference by time difference that differentiates images with time points before and after the shooting time. The human silhouette may be detected by a background difference method that differentiates the background image from the captured image. The human silhouette may be replaced by key points detected from the captured image using a trained neural network model. The behavior of the care recipient 80 can be detected from the temporal change of the posture (for example, standing position, sitting position, and lying position) of the care recipient 80 recognized based on the human silhouette. The behavior of the care recipient 80 may be detected from the relative positional relationship between the human silhouette and the furnishings in the living room such as the bed 90. For example, "getting up" may be detected as an action (state) in which the human silhouette crosses the area of the bed 90 preset as a coordinate area in the captured image.

[0041] The information regarding the behavior of the care recipient 80 detected by the detection unit 10 can be reflected in the care record of the care recipient 80. Further, the captured image captured by the camera 14 is stored in the storage unit 23.

[0042] <Server 20> FIG. 4 is a block diagram illustrating the hardware configuration of the server 20. The server 20 includes a control unit 21, a communication unit 22, and a storage unit 23, which are interconnected by a bus. The server 20 functions as a care recording device.

[0043] The care recording device according to the present embodiment acquires an image including a person (target person) whose behavior is to be estimated, determines the behavior of the person based on the image, and performs face authentication of the person. Further, the care recording device outputs information regarding the behavior of the person based on the determination result of the behavior and the authentication result of the person. Details of each component of the care recording device will be described later.

[0044] The control unit 21 includes a CPU, a RAM, a ROM, etc. (not shown). The control unit 21 controls each part of the server 20 and performs arithmetic processing according to the care record program. Details of the functions of the control unit 21 will be described later.

[0045] The communication unit 22 is constituted by an interface circuit for communicating with, for example, the detection unit 10, the fixed terminal 30, and the mobile terminal 40 via a LAN.

[0046] The storage unit 23 is constituted by an HDD, an SSD, etc., and stores an OS (Operating System), various programs such as a care record program, and various data. The various data includes, for example, care records for each care recipient 80, authentication information for the care staff 70 and the care recipient 80, information regarding the positions where the cameras 51 to 54 in the shared space photograph people, etc. Usually, the positions where the cameras 51 to 54 photograph people are considered to be the places where the cameras 51 to 54 are installed. In this case, the information regarding the above positions includes, for example, places in the shared space such as the cafeteria, the washroom, in front of the toilet, in front of the bathroom, the lobby, the corridor, etc. For example, in the storage unit 23, as the information regarding the above positions, it is registered that the cameras 51 to 54 are respectively associated with the cafeteria, the washroom, in front of the toilet, and in front of the bathroom (see FIG. 1).

[0047] <Fixed terminal 30> FIG. 5 is a block diagram illustrating the hardware configuration of the fixed terminal 30. The fixed terminal 30 is a PC (Personal Computer) and includes a control unit 31, a communication unit 32, a storage unit 33, a display unit 34, and an input unit 35. Since the basic configurations of the control unit 31, the communication unit 32, and the storage unit 33 are the same as those of the corresponding elements of the server 20, duplicate explanations are omitted.

[0048] The display unit 34 is composed of a display (e.g., a liquid crystal display) and displays various information. The display unit 34 can display, for example, the care records stored in the storage unit 23 of the control unit 21 or the care record drafts generated by the control unit 21. The input unit 35 is composed of a keyboard or a touch panel and accepts the operation input of various information by an administrator or the like.

[0049] The control unit 31 can accept, via the input unit 35, the input of a table in which the correspondence relationship between the unique ID for specifying the detection unit 10 and the care recipient 80 is registered, transmit it to the server 20, and store it in the storage unit 23.

[0050] Also, the control unit 31 can accept, via the input unit 35, a table in which the correspondence relationship between the unique ID for specifying the mobile terminal 40 and the care staff 70 is registered, transmit it to the server 20, and store it in the storage unit 23.

[0051] Also, the control unit 31 can accept, via the input unit 35, a table in which the correspondence relationship between the care staff 70 and the care recipients 80 in charge of the care staff 70 is registered, transmit it to the server 20, and store it in the storage unit 23.

[0052] Also, the control unit 31 can accept, via the input unit 35, the shift schedule or the like of the care staff 70, transmit it to the server 20, and store it in the storage unit 23.

[0053] In addition, care staff 70, technical staff, etc. associate the room numbers with the detection units 10 when the detection units 10 are installed in each room (living room) through the fixed terminal 30. Also, care staff 70, technical staff, etc. calibrate and specify the position information of the furnishings in the living room such as the bed 90, that is, the overhead contour information captured by the ceiling camera 14. Further, care staff 70, technical staff, etc. also associate the personal ID (name, identification number, etc.) of the care recipient 80 living in the facility with each room number. The fixed terminal 30 manages a list of care recipients including the personal ID of the care recipient 80, the room number, the name of the assigned staff, the affiliated unit, etc. The list of care recipients can be shared within the care record system 1.

[0054] <Mobile terminal 40> FIG. 6 is a block diagram showing the hardware configuration of the mobile terminal 40. The mobile terminal 40 includes a control unit 41, a wireless communication unit 42, a memory unit 43, an input / output display unit 44, an audio input / output unit 45, and a position detection unit 46, which are interconnected by a bus. The mobile terminal 40 can be configured by a portable communication terminal device such as a tablet computer, a smartphone, or a mobile phone. Since the basic configuration of the control unit 41 is the same as that of the control unit 21 of the server 20, duplicate explanations are omitted. Details of the functions of the control unit 41 will be described later.

[0055] The wireless communication unit 42 performs wireless communication with each device via the access point 61 or directly using wireless communication standards such as Wi-Fi and Bluetooth (registered trademark).

[0056] The memory unit 43 is composed of an SSD or an SD card and stores a mobile application corresponding to the care record system 1, various programs, and various data.

[0057] The input display unit 44 is constituted by, for example, a touch panel in which a touch sensor is superimposed on a display surface such as a liquid crystal. The input display unit 44 displays various information and accepts various inputs. The input display unit 44 can display, for example, the care record stored in the storage unit 23 of the control unit 21 or the care record draft generated by the control unit 21. Further, the input display unit 44 functions as a reception unit and accepts changes to the care record draft.

[0058] The voice input / output unit 45 is, for example, a speaker and a microphone, and enables a voice call by the care staff 70 with another mobile terminal 40 through the wireless communication unit 42. The voice input / output unit 45 enables a voice call by the care staff 70 with the care recipient 80 between the detection unit 10 through the server 20 by the wireless communication unit 42.

[0059] The position detection unit 46 is, for example, a GPS module or the like, and detects the position of the mobile terminal 40. As will be described later, the position of the mobile terminal 40 may be detected by the control unit 41 using signals transmitted from each beacon installed at a predetermined installation position.

[0060] <Functions of the control unit 21> FIG. 7 is a block diagram illustrating the functions of the control unit 21 of the care record device 20 shown in FIG. 4. As shown in the figure, the control unit 21 functions as an acquisition unit 211, an estimation unit 212, an authentication unit 213, a determination unit 214, and an output unit 215. FIG. 8 is a flowchart for explaining the processing procedure of the care record method according to an embodiment. The processing of the flowchart shown in the figure is realized by the CPU of the control unit 21 executing the care record program.

[0061] Further, FIG. 9 is a schematic diagram illustrating meal care in a shared space (cafeteria), and FIG. 10 is a schematic diagram illustrating an image of the shared space (cafeteria) taken by a camera. Further, FIG. 11 is a schematic diagram illustrating joint point estimation for a person included in the image. Further, FIG. 12 is a schematic diagram illustrating face authentication of a person. Further, FIG. 13 is a graph illustrating the time change of the position coordinates of the joint points of the care staff 70 and the care recipient 80. In the figure, the horizontal axis represents time, and the vertical axis represents the position coordinates of the joint points in the X-axis direction. Further, FIG. 14 is a diagram illustrating the action data of the care staff 70 in meal care, and FIG. 15 is a diagram illustrating the action data of the care recipient 80 in meal care. Further, FIG. 16 is a schematic diagram illustrating a recording table in meal care, and FIG. 17 is a schematic diagram illustrating the display of the generated care record draft.

[0062] [Example of Action Estimation in the Cafeteria] Hereinafter, the case of estimating the actions of a person in the cafeteria will be illustrated and described. As shown in FIG. 8, the acquisition unit 211 acquires an image including a person (step S101). For example, the care recipient 80 enters the cafeteria accompanied by the care staff 70 to have a meal and sits on the chair at the table. In the present embodiment, the camera 51 is installed on the ceiling or the wall surface so that, for example, a person sitting at the table in the cafeteria is included in the shooting range.

[0063] As shown in FIG. 9, the camera 51 shoots a person within the shooting range in the cafeteria. In the example shown in the figure, the case where the care staff 70 is providing meal care to the care recipient 80 within the shooting range of the camera 51 installed in the cafeteria is illustrated. The camera 51 shoots the care staff 70 and the care recipient 80 as the above-mentioned persons, and transmits the captured video to the control unit 21 of the server 20, and the acquisition unit 211 acquires an image including the care staff 70 and the care recipient 80.

[0064] Shooting by the camera 51 may be configured to be always performed during a predetermined time period (for example, from 9:00 to 22:00), or may be configured to be performed only while there are people in the cafeteria. The presence or absence of people in the cafeteria can be detected by, for example, a human sensor or the like.

[0065] In addition, the control unit 21 transmits the captured video of the cafeteria to the fixed terminal 30 and / or the mobile terminal 40 in response to requests from users such as administrators and care staff 70. As shown in FIG. 10, an administrator or the like can view the video of the cafeteria displayed on the fixed terminal 30. In addition, the care staff 70 can view the video of the cafeteria displayed on the mobile terminal 40.

[0066] Next, the estimation unit 212 estimates the joint points of the person (step S102). As shown in FIG. 11, the estimation unit 212 estimates the joint points of the persons 1 and 2 included in the image based on the distance image captured by the camera 51 in the cafeteria.

[0067] In the same figure, the estimated joint points are indicated by "〇". The joint points estimated by the estimation unit 212 include, for example, eyes (left and right), ears (left and right), nose, elbows (left and right), wrists (left and right), shoulders (left and right), neck, waist (left and right), pelvis, knees (left and right), and ankles (left and right). In this embodiment, the case of mainly focusing on the wrists and nose as the joint points of the person will be described. In the same figure, for the sake of simplicity of illustration, the illustration of joint points other than eyes (left and right), nose, elbows (left and right), wrists (left and right), shoulders (left and right), and neck is omitted.

[0068] Next, the authentication unit 213 performs face authentication of the person to identify the person (step S103). The authentication unit 213 authenticates a plurality of persons 1 and 2 for whom joint points have been estimated based on the image acquired by the acquisition unit 211. As shown in FIG. 12, the authentication unit 213 identifies that the plurality of persons 1 and 2 included in the above image are the pre-registered care staff S1 and the care recipient C.

[0069] More specifically, the authentication unit 213 acquires distance values to each part of a person's face (e.g., eyes (left and right), ears (left and right), nose, etc.) based on the distance image captured by the camera 51, and performs face authentication of the person. Alternatively, the authentication unit 213 may be configured to perform face authentication of the person based on the captured image captured by the visible light camera.

[0070] In this way, by performing face authentication of the person, the control unit 21 can identify who each person is who is acting in the shared space other than the living room of the care recipient 80. Further, by identifying the person in the shared space, the control unit 21 can individually estimate the activity amount of the care recipient 80 and the care amount (care time) of the care staff 70.

[0071] Next, the determination unit 214 determines the action of the person (step S104). The determination unit 214 determines the action of the person based on the temporal change in the position of the joint points of the person or the skeleton connecting the joint points, which is estimated by the estimation unit 212. The temporal change in the position of the joint points or the skeleton includes, for example, at least either the movement amount or the movement direction of the position of the joint points or the skeleton.

[0072] Further, the determination unit 214 may determine the action of the person based only on the temporal change in the position of the joint points or the skeleton of the person, or may determine the action of the person based on the temporal change in the position of the joint points or the skeleton of the person and the authentication result. That is, it is possible to determine the action of the person based on the image captured by the camera 51 after recognizing who the person is, or it is also possible to determine the action of the person based only on the above image without recognizing who the person is. By taking the authentication result into account, the determination unit 214 can accurately determine the respective actions of the care staff 70 and the care recipient 80 even when the temporal changes in the positions of the joint points of the care staff 70 and the care recipient 80 are similar.

[0073] As shown in FIG. 13, the determination unit 214 determines the actions of the care staff S1 and the care recipient C based on the change over time in the position of the joint point 71 of the wrist of the care staff S1 and the change over time in the position of the joint point 81 of the nose of the care recipient C. Alternatively, the determination unit 214 may be configured to determine the actions of the care staff S1 and the care recipient C based on the skeleton 73 connecting the joint point 71 of the wrist to the joint point 72 of the elbow of the care staff S1 and the change over time in the position of the joint point 81 of the nose of the care recipient C.

[0074] In the figure, the change over time in the position of the joint point 71 of the wrist corresponding to the actions of the care staff S1 in typical meal care and the change over time in the position of the joint point 81 of the nose of the care recipient C are illustrated.

[0075] Also, the determination unit 214 acquires the cafeteria as information regarding the position where the camera 51 photographs the care staff S1 and the care recipient C, and extracts candidates for the actions of the care staff S1 and the care recipient C based on the information. As actions of the care recipient 80 in the cafeteria, for example, (1) sitting on a chair at a predetermined table in the cafeteria, (2) starting to eat, (3) leaving the seat, (4) resuming eating, and (5) finishing eating are extracted. Hereinafter, specific examples of the relationship between these actions and the change over time in the position of each of the joint point 71 of the wrist of the care staff S1 and the joint point 81 of the nose of the care recipient C will be described.

[0076] (1) After entering the cafeteria, the care staff S1 and the care recipient C each sit on a chair at a predetermined table. Let the time when the care staff S1 and the care recipient C sit down be time T0. The movements (i.e., the change over time in position) of the care staff S1 and the care recipient C temporarily become smaller due to the action of sitting down as compared to just before sitting down. For example, the joint point 71 of the wrist of the care staff S1 stops at approximately position X1 from time T0 to time T1. Also, the joint point 81 of the nose of the care recipient C also stops in the vicinity of approximately the same position X2 from time T0 to T1.

[0077] After a temporary stationary state following the sitting-down motion, meal care starts. When the meal care begins, for example, care staff S1 scoops up food from a dish containing the food using a spoon or the like, moves it to near the mouth of care recipient C, and repeats the action of returning the empty spoon to the dish. By this action, for example, the joint point 71 of the wrist of care staff S1 repeatedly moves back and forth several times between approximately position X1 and position X2 in the X-axis direction from time T1 to T2. In FIG. 12, the movement of the position of joint point 71 can be represented as a waveform with X1 as the minimum value and X2 as the maximum value (X2 - X1 as the amplitude). That is, when joint point 71 is at X1, the spoon is in the dish, and when joint point 71 is at X2, the spoon is near the mouth of care recipient C. On the other hand, the joint point 81 of the nose of care recipient C stays near approximately the same position X2 from time T0 to T2.

[0078] (2) The determination unit 214 can determine, for example, the point in time when the spoon reaches the mouth of care recipient C, that is, the point in time when the joint point 71 of the wrist is closest to the joint point 81 of the nose of care recipient C, as the "start of meal". For example, the determination unit 214 can determine the point in time when joint point 71 first reaches position X2 after time T0 as the start of the meal.

[0079] Also, the determination unit 113 determines, for example, based on the change over time of the position of joint point 71 in the X-axis direction, that the action of care staff S1 is meal care. Also, the determination unit 113 determines, based on the change over time of the position of joint point 81 in the X-axis direction, that the action of care recipient C is eating.

[0080] As shown in FIGS. 14 and 15, when care recipient C starts eating as a specific action, the output unit 215 outputs action data as information regarding the action to the storage unit 23. The action data of care staff 70 can include, for example, the date and time when the action was performed by care staff 70, the identification information of the care staff 70 (care staff S1) who performed the action, the content of the action, and the like. The identification information of care staff 70 can be, for example, the personal ID, name, etc. of care staff 70. The content of the action can be, in the case of meal care, for example, sitting down, start of meal care, during meal care, leaving the seat, end of meal care, and the like.

[0081] In addition, the behavior data of the care recipient 80 may include, for example, the date and time when the behavior of the care recipient 80 occurred, the identification information of the care recipient 80 (care recipient C) who performed the behavior, the content of the behavior, and the like. The identification information of the care recipient 80 may be, for example, the personal ID of the care recipient 80, the name, and the like. The content of the behavior may be, for example, seating, start of meal, during meal, leaving the table, end of meal, etc. in meal care.

[0082] In addition, the authentication unit 213 assigns a transaction (tracking) ID to the pair of the care staff 70 and the care recipient 80 within the shooting range of the camera 51. The transaction ID is assigned different identifiers according to the presence or absence of care and the type of care for each pair. For example, for pairs where no care is being provided, identifiers such as 1, 2, 3,... are assigned. Until the care staff S1 and the care recipient C are seated, "1" is assigned as the transaction ID. Further, if there is another pair in the cafeteria, for example, "2" or "3" is assigned. These transaction IDs are normal transaction IDs that are not recorded in the care record.

[0083] In addition, when the care recipient 80 starts a predetermined behavior that should leave a record in the care record such as a meal or a bath, the authentication unit 213 switches the normal transaction ID to a transaction ID for recording and manages it. For example, when a predetermined behavior of the care recipient 80 starts, the authentication unit 213 switches to a transaction ID for the predetermined behavior and holds the same transaction ID for the predetermined behavior until the predetermined behavior ends. Also, after the predetermined behavior ends, the authentication unit 213 switches from the transaction ID for the predetermined behavior to the normal transaction ID and continues the management.

[0084] For example, when meal care is started, the authentication unit 213 assigns, for example, M1, M2, M3, … as identifiers for meal care to the pair for which meal care is being provided. For example, when meal care is started for the care recipient C by the care staff S1, “M1” is assigned as the transaction ID. Similarly, when meal care is being provided by another pair of care staff 70 and care recipient 80 in the cafeteria, for example, “M2” or “M3” is assigned.

[0085] Also, although not shown in FIGS. 9, 10, etc., assume that another care recipient E is seated immediately to the right of the care staff S1, and the care recipient E is receiving meal care from the care staff S1 in the same manner as the care recipient C. In this case, meal care is provided simultaneously to the care recipients C and E by the same care staff S1. Therefore, the same identifier “M1” as that of the pair of the care staff S1 and the care recipient C is assigned as the transaction ID to the pair of the care staff S1 and the care recipient E.

[0086] Returning to FIG. 13 again, during the time period (3) T2 - T3, for example, the care recipient C may leave the seat by going to the toilet or returning to the living room to pick up something forgotten. Also, since the care staff S1 also accompanies the care recipient C, the care staff S1 may leave the seat. In this case, since the care staff S1 and the care recipient C move outside the shooting range of the camera 51, the estimation of the joint points 71, 81 of the care staff S1 and the care recipient C is not performed. Based on the fact that there is no change over time in the positions of the joint points 71, 81, the determination unit 214 determines the actions of the care staff S1 and the care recipient C during this period as leaving the seat.

[0087] (4) At time T3, the care staff S1 and the care recipient C return to the cafeteria, and meal care resumes. The care staff S1 repeatedly performs the operation of scooping up the food from the dish containing the food using a spoon or the like, bringing it to the mouth of the care recipient C, and returning the empty spoon to the dish. By this operation, for example, the joint point 71 of the wrist of the care staff S1 reciprocates many times between approximately position X1 and position X2 in the X-axis direction from time T3 to T4. In addition, regarding the temporal change in the position of the joint point 71 of the wrist of the care staff S1 and the joint point 81 of the nose of the care recipient C when the care staff S1 and the care recipient C are seated at the restart of meal care, the description is omitted because it follows the same process as in (1) above. The output unit 215 outputs action data as information regarding the action to the storage unit 23 when the care recipient C starts (restarts) eating as a specific action.

[0088] (5) The determination unit 214 determines "meal finished" when a predetermined time ta has elapsed after the joint point 71 of the wrist has returned to the vicinity of position X1 (time T4). The output unit 215 outputs action data as information regarding the action to the storage unit 23 when the care recipient C finishes eating as a specific action.

[0089] Note that in the above example, the case where the care staff S1 enters the cafeteria with the care recipient C has been illustrated and described. However, the present invention is not limited to such a case. For example, even in the case where the care recipient C enters the cafeteria alone and then the care staff S1 enters and performs meal care for the care recipient C, the actions of the care staff S1 and the care recipient C can be determined in the same way.

[0090] Next, the determination unit 214 identifies the content of the care (step S105). The determination unit 214 identifies the content of the care provided by the care staff S1 to the care recipient C based on the determination results of the actions of the care staff S1 and the care recipient C. In the example shown in FIG. 12, it is identified that the content of the care is meal care. Usually, "meal start" and "meal end" are paired. The determination unit 214 can determine, for example, that the content of the care provided by the care staff S1 to the care recipient C is meal care when "meal start" and "meal end" are detected. Alternatively, the determination unit 214 may be configured to determine, for example, that the content of the care provided by the care staff S1 to the care recipient C is meal care when "meal start" is detected.

[0091] Next, the output unit 215 outputs information regarding the actions of persons 1 and 2 (care staff S1 and care recipient C) (step S106). More specifically, the output unit 215 outputs information regarding the actions of the care staff S1 and the care recipient C based on the authentication result by the authentication unit 213 and the determination result by the determination unit 214. The information regarding the actions may be information regarding the care including the identified content of the care.

[0092] The output unit 215 generates information regarding the care based on the action data with the transaction ID "M1" and stores it in the storage unit 23. For example, as shown in FIG. 16, the output unit 215 creates a recording table as information regarding the care based on the action data of the care staff S1 (FIG. 14), the action data of the care recipient C (FIG. 15), and the action data of the care recipient E (not shown).

[0093] The recording table may include, for example, items such as the date and time when the care was provided, the identification information of the care staff 70 who provided the care and the care recipient 80 who received the care, the content of the care, the start time, the end time, and the interruption time. The start time and the end time respectively correspond to the meal care start time and the meal care end time in FIG. 14. The interruption time represents, for example, the time when the care recipient 80 interrupted the meal due to leaving the seat or the like. Also, the time when the care was provided is the same as the meal care start time.

[0094] The output unit 215 creates a provisional care record (hereinafter referred to as "care record draft") based on the generated record table (information regarding care). The output unit 215 converts the content of the created record table to match the format of a predetermined care record to generate a care record draft, and outputs the care record draft to the portable terminal 40. As shown in FIG. 17, the portable terminal 40 displays the care record draft on the display.

[0095] Next, the portable terminal 40 accepts changes to the care record draft (step S107). The care staff S1 checks the content of the care record draft, and if it is correct, registers it as the content of the care record (by pressing the send button).

[0096] Also, if there are items (changes) to be corrected or added to the content of the care record draft, the care staff S1 inputs the items to be corrected or added to the portable terminal 40, and the portable terminal 40 accepts the changes to the care record draft. Examples of items to be added include the amount of food and the state of the care recipient C during meals.

[0097] Thus, in the process of the flowchart shown in FIG. 8, the control unit 21 acquires an image including a person, performs face authentication of the person, and identifies the person. Further, the control unit 21 determines the action of the person based on the above image, and outputs information regarding the action of the person based on the authentication result and the determination result. The control unit 21 can update the content of the care records of the care recipients C and E using the information regarding the actions of the care staff S1 and the care recipients C and E. On the other hand, in the care record system 1 of the present embodiment, the generated care record draft is not directly registered as the care record, but is configured such that a care record draft whose content has been changed by the care staff 70 is registered as needed. Thereby, while reducing the burden of inputting the care record by the care staff 70, the accuracy of the content described in the care record can be improved or maintained.

[0098] Also, since the person photographed in the shared space is identified by face authentication, the activity amount of the care recipient 80 and the care amount of the care staff 70 can be estimated.

[0099] In addition, since the control unit 21 can acquire the positions of the joint points of the bodies of the care staff 70 and the care recipient 80 in the shared space in time series, it can determine who is facing whom.

[0100] In addition, the control unit 21 can obtain information on the daily actions of the care recipient 80 by synthesizing the determination result of the actions of the care recipient 80 in the shared space and the determination result of the actions in the living room. Furthermore, the control unit 21 can analyze the daily rhythm of the care recipient 80 based on the information on the daily actions of the care recipient 80 and visualize the analysis result in the form of a graph, table, etc.

[0101] [Example of Action Estimation of Persons Before and After Bathing Care] Hereinafter, a case of estimating the start and end of bathing care by estimating the actions of a person before and after bathing care will be illustrated and explained as an example.

[0102] FIG. 18 is a graph illustrating the time change of the position coordinates of the joint points of the care staff 70 and the care recipient 80 before and after bathing care. In the figure, the horizontal axis is time, and the vertical axis is the position coordinate of the joint point in the Y-axis direction. FIG. 19 is a diagram illustrating the action data of the care staff 70 before and after bathing care. FIG. 20 is a diagram illustrating the action data of the care recipient 80 before and after bathing care. FIG. 21 is a diagram illustrating the recording table before and after bathing care.

[0103] For example, in order to take a bath, the care recipient 80 enters the bathroom accompanied by the care staff 70, receives bathing care from the care staff 70 in the bathroom, and then exits the bathroom accompanied by the care staff 70 again. The camera 52 is installed in front of the bathroom and photographs the persons entering and leaving the bathroom. The camera 52 photographs the care staff 70 and the care recipient 80 as the above-mentioned persons, and transmits the photographed image to the control unit 21 of the server 20, and the acquisition unit 211 acquires the image including the care staff 70 and the care recipient 80.

[0104] The estimation unit 212 estimates the joint points of the two persons included in the image based on the image captured by the camera 52 in front of the bathroom.

[0105] The authentication unit 213 authenticates the two persons whose joint points are estimated based on the image acquired by the acquisition unit 211, and specifies, for example, that one person is the care staff S2 and the other person is the care recipient D.

[0106] The determination unit 214 determines the actions of the care staff S2 and the care recipient D based on the change over time in the position of the joint point of the wrist of the care staff S2 and the change over time in the position of the joint point of the wrist of the care recipient D. Alternatively, the determination unit 214 may be configured to determine the actions of the care staff S2 and the care recipient D based on the change over time in the position of the skeleton connecting the joint point of the wrist to the joint point of the elbow of the care staff S2 and the change over time in the position of the skeleton connecting the joint point of the wrist to the joint point of the elbow of the care recipient D.

[0107] FIG. 18 illustrates the change over time in the position of the joint point of the wrist corresponding to the actions of the care staff S2 before and after the bathing care, and the change over time in the position of the joint point of the wrist of the care recipient D before and after the bathing care.

[0108] Further, the determination unit 214 acquires in front of the bathroom as information on the position where the camera 52 captures the care staff 70 and the care recipient 80, and extracts candidates for the actions of the care staff 70 and the care recipient 80 based on the information. As candidates for the actions of the care recipient 80 before and after the bathing care, the determination unit 214 extracts, for example, (1) walking, (2) entering the bathroom, and (3) leaving the bathroom.

[0109] In the example shown in the figure, for example, the care recipient D walks slowly towards the bathroom while being supported by the care staff S2. The positions of the articulation points of the wrists of the care recipient D and the care staff S2 vibrate slightly in the vertical direction (Y-axis direction) as they walk. The movement of the position of the articulation point can be represented as a waveform with Y1 as the minimum value and Y2 as the maximum value (Y2 - Y1 being the amplitude). The determination unit 113 determines that the actions of the care staff S2 and the care recipient D are walking based on the change over time in the position of the articulation points of the wrists of the care staff S2 and the care recipient D in the Y-axis direction.

[0110] As shown in FIGS. 19 and 20, when the care recipient D starts walking as a specific action, the output unit 215 outputs action data as information related to the action to the storage unit 23.

[0111] Subsequently, from when the care staff S2 and the care recipient D enter the bathroom until they leave, the care staff S2 and the care recipient D are outside the shooting range of the camera 51. Therefore, the camera 52 cannot shoot the care staff S2 and the care recipient D. For this reason, the estimation unit 212 cannot estimate the articulation points of the care staff S2 and the care recipient D. In the figure, the position of the articulation point is represented as blank from when the care staff S2 and the care recipient D enter the bathroom until they leave. It is estimated that the care recipient D is receiving bathing care from the care staff S2 from when the care staff S2 and the care recipient D enter the bathroom until they leave.

[0112] After the care recipient D leaves the bathroom, the care recipient D walks slowly towards the living room while being supported by the care staff S2 in the same manner as when going towards the bathroom. The determination unit 214 determines that the actions of the care staff S2 and the care recipient D are walking based on the change over time in the position of the articulation points of the wrists of the care staff S2 and the care recipient D in the Y-axis direction. The output unit 215 outputs action data as information related to the action to the storage unit 23 when the care recipient D starts walking as a specific action.

[0113] Based on the determination results of the actions of the care staff S2 and the care recipient D, the determination unit 214 identifies the content of the care provided by the care staff S2 to the care recipient D. In the example shown in FIG. 18, it is identified that the content of the care is bathing care.

[0114] As shown in FIG. 21, the output unit 215 creates a recording table as care-related information based on the action data of the care staff S2 (FIG. 19) and the action data of the care recipient D (FIG. 20).

[0115] The output unit 215 creates a care record draft based on the generated recording table (care-related information). The output unit 215 converts the content of the created recording table to match a predetermined care record format to generate a care record draft, and outputs the care record draft to the portable terminal 40. The portable terminal 40 displays the care record draft on the display.

[0116] The portable terminal 40 accepts changes to the care record draft. Since the changes to the care record draft are the same as the content described in the above [Example of Action Estimation in the Dining Hall], detailed description is omitted.

[0117] Also, above, examples of action estimation of people before and after bathing care have been described, but actions of people before and after excretion care can be estimated in the same way.

[0118] [Recording Method of Care Records for the Entire Facility] FIG. 22 is a diagram illustrating a recording method of care records for the entire facility. Examples of care content in the facility include, for example, meals, bathing, excretion, oral care, medication administration, assistance (outside / inside the room), recreation, hydration, monitoring, etc. Among these, the action determination of people in meal care and bathing (excretion) care is as described above. In oral care, medication administration, assistance (outside / inside the room), recreation, hydration, monitoring, etc., face authentication can also be used as a method for discriminating people. Based on the authentication result of the person and the estimation result of the joints or skeleton of the person based on the captured image by the camera, the action of the person can be estimated.

[0119] Regarding assistance (indoors) and monitoring, as a method for detecting a person, a sensor (detection unit 10) may be used. Further, instead of the authentication unit 213, for example, based on the difference in the stride and walking speed of a person from the video captured by the camera 14, the care staff 70 and the care recipient 80 may be discriminated.

[0120] The configuration of the system described above has described the main configuration in explaining the features of the above-described embodiment, and is not limited to the above-described configuration, and various modifications can be made within the scope of the claims.

[0121] For example, in the above example, the case where the server 20 of the care record system 1 functions as a care record device has been described, but the present invention is not limited to such a case. The care record device may be implemented in a personal computer, server, cloud, etc. independent of the care record system 1, and may be configured to acquire images in a shared space or in a living room from devices used in a care facility through a network.

[0122] Also, in the above example, for the sake of explanation, the case where the actions of the care staff 70 and the care recipient 80 are determined based on the temporal change in the position of the joint points in the X-axis direction or the Y-axis direction has been described, but the present invention is not limited to such a case. For example, the determination unit 214 may be configured to determine the actions of the care staff 70 and the care recipient 80 based on the temporal change in the position of the joint points in at least any one of the three directions of the X-axis, Y-axis, and Z-axis directions.

[0123] Also, in the above example, the case where the estimation unit 212 estimates joint points based on the distance image of the body has been described. However, the present invention is not limited to such a case. For example, the cameras 51 to 54 are, for example, visible light cameras, and the captured image can be transmitted to the estimation unit 212. The estimation unit 212 can estimate the joint points of the person 1 and the person 2 included in the captured image based on the captured image.

[0124] For example, the estimation unit 212 can detect a person rectangle that includes a person from a captured image using a learned model of a neural network that has been trained to estimate the person rectangle. Then, the estimation unit 212 can detect the joint points of a person using a learned model of a neural network that has been trained to estimate joint points from the person rectangle. As a learned model for estimating a person rectangle from a captured image, for example, there is a model of RPN (Region Proposal Network). Also, as a learned model for detecting the joint points of a person from a person rectangle, for example, there are models such as Deep Pose, CNN (Convolution Neural Network), and Res Net.

[0125] Also, in the above-described flowchart, steps other than those shown in the flowchart may be included, or some steps may not be included. Also, the order of the steps is not limited to the above-described embodiment. Furthermore, each step may be executed as one step in combination with other steps, may be executed included in other steps, or may be divided into a plurality of steps and executed.

[0126] Also, the means and methods for performing the various processes in the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided by a computer-readable recording medium such as a USB memory or a DVD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a storage device such as a hard disk. Also, the above program may be provided as a single application software or may be incorporated into the software of the device as one function.

Description of Reference Numerals

[0127] 1 Nursing care recording system, 10 Detection unit, 11 Control unit, 12 Communication unit, 13 Memory unit, 14 Camera, 15 Body movement sensor, 16 Care call unit, 20 Server, 21 Control unit, 211 Acquisition unit, 212 Estimation unit, 213 Authentication unit, 214 Judgment unit, 215 Output unit, 22 Communication unit, 23 Memory unit, 30 Fixed terminal, 31 Control unit, 32 Communication unit, 33 Memory unit, 34 Display unit, 35 Input unit, 40 Mobile terminal, 41 Control unit, 42 Wireless communication unit, 43 Memory unit, 44 Input display unit, 45 Audio input / output unit, 46 Position detection unit, 51 - 54 Camera, 60 Network, 70 Care staff, 80 Care recipient, 90 Bed.

Claims

1. An acquisition unit that acquires an image including a person; An authentication unit that performs face authentication on the person to identify the person; A determination unit that determines the action of the person based on the image; An output unit that outputs information regarding the action of the person based on the authentication result by the authentication unit and the determination result by the determination unit; A care record system comprising the above.

2. Further comprising an estimation unit that estimates the joint points of the person based on the image, The determination unit determines the action of the person based on the joint points estimated by the estimation unit or the temporal change in the position of the skeleton connecting the joint points. The care record system according to Claim 1.

3. The authentication unit performs authentication of the person based on the image. The care record system according to Claim 1 or 2.

4. The authentication unit authenticates that a plurality of persons included in the image are pre-registered caregivers and care recipients. The care record system according to Claim 3.

5. Based on the determination result, the determination unit specifies the content of care provided by the caregiver to the care recipient and outputs it as information regarding the action of the person. The care record system according to Claim 4.

6. The output unit outputs information regarding the action of the person to the terminal of the caregiver as a care record of the care recipient. The care record system according to Claim 5.

7. Further comprising a reception unit that receives changes to the care record. The care record system according to Claim 6.

8. Further comprising a storage unit that stores the care record of the care recipient, The output unit updates the content of the care record using the information regarding the action of the person. The care record system according to Claim 4.

9. Further comprising a photographing unit that photographs the person, The determination unit acquires information regarding the position where the photographing unit photographs the person, and extracts candidates for the action of the person based on the information. The care record system according to Claim 1 or 2.

10. When the person starts a specific action, the output unit outputs information regarding the action. The care record system according to Claim 1 or 2.

11. The determination unit determines the action of the person based on the image and the authentication result. The care record system according to Claim 1 or 2.

12. Step (a) of acquiring an image including a person; Step (b) of performing face authentication on the person to identify the person; Step (c) of determining the actions of the person based on the image; Step (d) of outputting information regarding the actions of the person based on the authentication result in step (b) and the determination result in step (c), the nursing care recording method comprising the steps.

13. Procedure (a) of acquiring an image including a person; Procedure (b) of performing face authentication of the person to identify the person; Procedure (c) of determining the actions of the person based on the image; A nursing care recording program for causing a computer to execute a process including procedure (d) of outputting information regarding the actions of the person based on the authentication result in procedure (b) and the determination result in procedure (c).

Images