Care information recording device and care information recording method

Abstract

To automate the recording of implementation status of services provided to users in nursing care facilities to reduce the burden on staff.SOLUTION: A management server 2 records and manages the implementation status of services provided to users in nursing care facilities. In particular, the management server identifies an object person as a user based on camera images of the object person, acquires movement information representing the movement of a predetermined part of the object person's body based on the camera images, recognizes the motion of the object person based on a plurality of continuous camera images, identifies a service provided to the object person and its implementation period based on the movement information and the recognition result of the motion, and generates service implementation information including user identification information, service identification information, and service implementation period information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a care information recording device and a care information recording method for recording and managing the implementation status of services provided to users in a care facility. [Background technology]

[0002] Nursing care facilities provide users with various services such as functional training and meals. These services are provided according to a pre-established schedule, but are not always carried out exactly as scheduled. For this reason, it is desirable to record the implementation status of the services provided to users.

[0003] As a technology related to recording the implementation status of services in such nursing care facilities, a technology has been known that collects information on the implementation status of services provided to users by recognizing the user's actions (waking up, falling asleep, going to the toilet, etc.) through image analysis of camera images taken of the user (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 142451 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, the user's movements that can be recognized through image analysis only relate to part of the services provided to the user at the care facility. Therefore, information on the implementation status of the remaining services must be collected manually by staff using tablets or other devices. This poses a problem in that it is not possible to sufficiently reduce the burden on staff.

[0006] Therefore, the main object of the present invention is to provide a care information recording device and a care information recording method that can automate the recording of the implementation status of services provided to users in care facilities and reduce the burden on staff. [Means for solving the problem]

[0007] The care information recording device of the present invention is a care information recording device that uses a processor to execute a process of recording and managing the implementation status of services provided to users in a care facility, and the processor identifies a target person as a user based on a camera image of the target person. and obtain user identification information, Movement information representing the movement of a predetermined part of the body of the target person based on the camera image. Based on the change in the amount of change in the target person, a movement period associated with the provision of the service to the target person is obtained, A plurality of consecutive camera images The clip video consisting of the above is input into the image recognition model in sequence, and action class information is sequentially acquired, and the service identification information and recognition score included in the action period and the action class information are sequentially acquired. Services provided to the subject person pursuant to Types of and the period of implementation; The user identification information and information regarding the type of service and its implementation period are The service execution information including the service execution information is generated.

[0008] Further, a care information recording method of the present invention is a care information recording method that causes an information processing device to perform a process of recording and managing the implementation status of services provided to users in a care facility, The information processing device includes: Identify the target person as a user based on the camera image of the target person and obtain user identification information, Movement information representing the movement of a predetermined part of the body of the target person based on the camera image. Based on the change in the amount of change in the target person, a movement period associated with the provision of the service to the target person is obtained, A plurality of consecutive camera images The clip video consisting of the above is input into the image recognition model in sequence, and action class information is sequentially acquired, and the service identification information and recognition score included in the action period and the action class information are sequentially acquired. Services provided to the subject person pursuant to Types of and the period of implementation; The user identification information and information regarding the type of service and its implementation period are The service execution information including the service execution information is generated. [Effects of the Invention]

[0009] According to the present invention, it is possible to accurately identify the services actually provided to users and the period during which they were provided, with respect to various services provided at a nursing care facility. This makes it possible to automate the recording of the implementation status of services provided to users at the nursing care facility and reduce the burden on staff. [Brief explanation of the drawings]

[0010] [Figure 1] Overall configuration diagram of a nursing care service management system according to the present embodiment [Figure 2] Block diagram showing the general configuration of the management server [Figure 3] Flow diagram showing the procedure for acquiring person information performed by the management server [Figure 4] Flow diagram showing the procedure for motion information acquisition processing performed by the management server [Figure 5] An explanatory diagram showing an overview of the action recognition process performed on the management server. [Figure 6] Flow diagram showing the procedure for action recognition processing performed on the management server [Figure 7] An explanatory diagram showing an overview of the integrated judgment process performed on the management server [Figure 8] Flow diagram showing the procedure for the integrated judgment process performed on the management server [Figure 9] FIG. 10 is an explanatory diagram showing the contents of service implementation information managed by the management server. [Figure 10] An explanatory diagram showing the service implementation status confirmation screen displayed on the management terminal DETAILED DESCRIPTION OF THE INVENTION

[0011] The first invention made to solve the above problem is a care information recording device that uses a processor to execute a process of recording and managing the implementation status of services provided to users in a care facility, and the processor identifies a target person as a user based on a camera image of the target person. and obtain user identification information, Movement information representing the movement of a predetermined part of the body of the target person based on the camera image. Based on the change in the amount of change in the target person, a movement period associated with the provision of the service to the target person is obtained,A plurality of consecutive camera images The clip video consisting of the above is input into the image recognition model in sequence, and action class information is sequentially acquired, and the service identification information and recognition score included in the action period and the action class information are sequentially acquired. Services provided to the subject person pursuant to Types of and the period of implementation; The user identification information and information regarding the type of service and its implementation period are The service execution information including the service execution information is generated.

[0012] This makes it possible to accurately identify the various services actually provided to users at nursing care facilities and the period during which they were provided, thereby automating the recording of the status of services provided to users at nursing care facilities and reducing the burden on staff.

[0013] In addition, in a second invention, the processor is configured to identify the target person by face matching between the target person detected from the camera image and a user registered in advance.

[0014] This allows a target person detected from a camera image to be accurately identified as the user.

[0015] In addition, a third invention is configured such that the processor estimates positions of the joints of the target person based on the camera image, and acquires the movement information including joint position information as the estimation result.

[0016] This makes it possible to accurately acquire movement information that represents the movement of a predetermined part of the target person's body.

[0017] In addition, a fourth aspect of the present invention is a method for processing a plurality of data, the method comprising: service plan information relating to a schedule for providing a service to the user is read from the storage unit; The service plan information and the service execution information are output in association with each other.

[0018] This allows the administrator to easily check whether the service is being provided to the user as planned.

[0019] In a fifth aspect of the present invention, the processor is By type of service Implementation period and the service plan information included A planned period is associated with each type of service, and overlapping periods and integrated periods are calculated in the corresponding relationship between the implementation period and the planned period for each type of service, and an index representing the ratio of the overlapping periods to the integrated periods is calculated for each type of service.and comparing the index with a predetermined threshold value for each Types of services below the threshold are displayed in a distinguishable manner.

[0020] This allows the administrator to easily identify services for which there is a significant discrepancy between the implementation period and the planned period.

[0021] In addition, a sixth aspect of the present invention is a method for processing a personal information stored in a computer system, the method comprising: Service Type As link information for displaying the camera image corresponding to the file name is added to the service execution information.

[0022] This allows the administrator to check in detail the status of the service actually being provided to the user. In this case, camera images included in the implementation period may be played back as a video.

[0023] A seventh invention is a care information recording method for causing an information processing device to perform a process of recording and managing the implementation status of services provided to users in a care facility, the method comprising: The information processing device includes: Identify the target person as a user based on the camera image of the target person and obtain user identification information, Movement information representing the movement of a predetermined part of the body of the target person based on the camera image. Based on the change in the amount of change in the target person, a movement period associated with the provision of the service to the target person is obtained, A plurality of consecutive camera images The clip video consisting of the above is input into the image recognition model in sequence, and action class information is sequentially acquired, and the service identification information and recognition score included in the action period and the action class information are sequentially acquired. Services provided to the subject person pursuant to Types of and the period of implementation; The user identification information and information regarding the type of service and its implementation period are The service execution information including the service execution information is generated.

[0024] According to this, similar to the first invention, it is possible to automate the recording of the implementation status of services provided to users in a care facility, thereby reducing the burden on staff.

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0026] FIG. 1 is a diagram showing the overall configuration of a nursing care service management system according to this embodiment.

[0027] The nursing care service management system (nursing care information recording system) records and manages the implementation status of services provided to users in nursing care facilities. The nursing care service management system includes a camera 1, a management server 2 (nursing care information recording device, information processing device), and a management terminal 3 (administrator device). The camera 1, management server 2, and management terminal 3 are connected via a network.

[0028] Camera 1 is installed at an appropriate location within the care facility and captures images of users receiving services at the care facility.

[0029] The management server 2 acquires camera images of users from the camera 1 and performs processes such as person detection and motion recognition based on the camera images to generate and manage information related to the implementation status of services provided to users at the nursing care facility. The management server 2 is configured with a PC or the like. The management server 2 may be installed in the nursing care facility or may be a cloud computer.

[0030] The management terminal 3 is used by the care facility manager to view information managed by the management server 2 and input required information. The management terminal 3 is configured as a PC, a tablet terminal, or the like.

[0031] The services provided to users at nursing care facilities include, for example, functional training (walking training, upper body exercise, full body exercise, etc.), meals, and recreation (karaoke, flower arranging, games, etc.).

[0032] Next, we will explain the general configuration of the management server 2. Figure 2 is a block diagram showing the general configuration of the management server 2.

[0033] The management server 2 includes a communication unit 11, a storage unit 12, and a processor 13.

[0034] The communication unit 11 communicates between the camera 1 and the management terminal 3 .

[0035] The storage unit 12 stores programs executed by the processor 13. The storage unit 12 also stores person information, movement information, action class information, and service implementation information generated by the processor 13 for each user.

[0036] The processor 13 performs various processes by executing programs stored in the storage unit 12. In this embodiment, the processor 13 performs image acquisition processing, person information acquisition processing, movement information acquisition processing, action recognition processing, integrated determination processing, information management processing, and the like.

[0037] In the image acquisition process, the processor 13 acquires camera images (frames) at each time received from the camera 1 via the communication unit 11. Information such as the image ID (file name), the camera ID of the camera that captured the image, and the time of capture (timestamp) is added to the camera image. The timestamp may also be set in the file name.

[0038] In the person information acquisition process, processor 13 detects a person from the camera image and acquires information about the person. Specifically, first, a person is detected from the camera image (person detection process). At this time, a person ID is assigned to the detected person, and a rectangular person frame (person area) surrounding the person is set in the camera image, and position information (coordinates) of the person frame on the camera image is acquired.

[0039] In addition, in the person information acquisition process, processor 13 extracts feature information of a person from the camera image and tracks the person detected from the camera image based on the feature information of the person (person tracking process). Specifically, if the person detected from the camera image is the same person as a person previously detected, the same person ID is assigned to the person and the people are associated with each other.

[0040] In the person information acquisition process, when the processor 13 first detects a person from the camera image, it performs face matching to identify the person as the user (face matching process). At this time, it cuts out a face image of the target person from the current camera image and extracts facial feature information of the target person from the face image. Then, it matches the facial feature information of the target person with facial feature information of pre-registered users to identify the user who is the target person detected from the current camera image.

[0041] In addition, in the person information acquisition process, the processor 13 generates and outputs person information. The person information includes a person ID, a user ID, and position information (coordinates) of a person frame on a camera image.

[0042] Here, face matching processing is performed when a new person tracking process is started, and thereafter, the face matching result (user ID) is carried over by re-matching by person matching performed in the person tracking process. Also, in the person tracking process, whether or not it is the same person is determined by person matching based on feature information of the person's entire body (mainly clothing). This makes it possible to track people even in situations where it is not possible to photograph the person's face from the front.

[0043] In addition, while a user is receiving services at a care facility, a staff member from the care facility is often present near the user, but this staff member is not determined to be the user through facial matching and is therefore excluded from processing.

[0044] In this embodiment, the person who will be the user is identified by face matching, but this person identification is not limited to face matching. For example, an image of a name tag worn by the person may be extracted from a camera image, and person identification information such as the name may be obtained from the image of the name tag by image recognition. Furthermore, the user may be provided with a wireless tag (RFID tag) or the like that is associated with the user in advance, so that the user's location can be identified.

[0045] In the motion information acquisition process, processor 13 generates motion information representing the motion of each part of the person's body detected from the camera image. At this time, based on the position information of the person frame acquired in the person information acquisition process, a rectangular person frame area is cut out from the camera image to acquire a person image. Next, based on the person image, the positions of the person's joints are estimated, and position information (coordinates) of the joints is acquired as an estimation result (joint estimation process). Next, motion information is generated. The motion information includes a person ID and position information (coordinates) of each joint.

[0046] In the action recognition process, processor 13 recognizes a person's actions based on camera images (frames). First, a plurality of consecutive camera images (frames) are integrated to generate a clip video of a predetermined time (for example, 1 second or 2 seconds) (clip video generation process). Next, the clip video is input to an image recognition model (machine learning model) for action recognition, and an action class is output from the image recognition model as a recognition result.

[0047] In the action recognition process, the processor 13 generates action class information for each time point for the target person. This action class information includes position information of the person frame on the camera image, the name of the action class (class ID), and the recognition score of the action class (a numerical value indicating the likelihood of the recognition result).

[0048] In the integrated judgment process, the processor 13 performs an integrated judgment by combining the movement information obtained in the movement information generation process and the movement class information obtained in the movement recognition process, thereby identifying the service provided to the user.

[0049] At this time, first, a period (movement period) during which some movement is occurring in the person's body is estimated based on the movement information acquired in the movement information generation process (movement period acquisition process). Next, movement class information for each time included in the movement period is extracted from the movement class information of the target person stored in the storage unit 12 (movement class information extraction process). Next, based on the extracted movement class information, i.e., the movement class information for each time included in the movement period, the type of service provided to the target person is identified, and the period during which the service is provided (start time and end time) is identified (service identification process).

[0050] In the integrated determination process, the processor 13 generates service implementation information, which includes a user ID, a service ID, and information relating to the implementation period (start time and end time) of the service.

[0051] In the information management processing, the processor 13 generates service plan information for each user in response to input operations by staff using the management terminal 3. The service plan information relates to a schedule for providing services to users. In addition, in the information management processing, the processor 13 causes the management terminal 3 operated by the manager to display a screen that presents the manager with the service plan information and service implementation information for each person, as well as information obtained by statistically processing that information, in response to a request from the management terminal 3 operated by the manager. In addition, in the information management processing, the processor 13 outputs the service plan information and service implementation information for each person, as well as information obtained by statistically processing that information, as data in a predetermined format that is used when creating various report documents.

[0052] Next, a description will be given of the personal information acquisition process performed by the management server 2. Fig. 3 is a flowchart showing the steps of the personal information acquisition process.

[0053] The management server 2 performs a process (person information acquisition process) to acquire information about a person based on the camera image received from the camera 1. This person information acquisition process is executed every time the flow shown in FIG. 3 receives a camera image (frame) periodically transmitted from the camera 1 at each time.

[0054] In the person information acquisition process, as shown in FIG. 3, first, the processor 13 acquires the current camera image received from the camera 1 via the communication unit 11 (image acquisition process) (ST101).

[0055] Next, processor 13 detects a person from the current camera image (person detection process) (ST102). At this time, a rectangular person frame is set in the camera image, and position information (coordinates) of the person frame on the camera image is acquired.

[0056] Next, processor 13 extracts characteristic information of the person from the current camera image, and tracks the person detected from the camera image based on the characteristic information of the person (person tracking process) (ST103).

[0057] Next, processor 13 determines whether or not a person has been detected from the current camera image by the person detection process (ST104). Note that even if a person has not been detected from the current camera image, the process continues because there may be a person being tracked.

[0058] Here, if a person is detected from the current camera image (Yes in ST104), then processor 13 determines whether the target person detected from the current camera image is a person already being tracked (ST105).

[0059] If the target person detected from the current camera image is not a person already being tracked, that is, if the target person is detected for the first time and a new person tracking process is started (No in ST105), processor 13 performs face matching on the target person and identifies the target person as a user (ST106). At this time, facial feature information of the target person is extracted from the current camera image, and the facial feature information of the target person is matched with facial feature information of pre-registered users to identify which user the target person is.

[0060] Next, the processor 13 generates and outputs person information (ST107). The person information includes a person ID, a user ID, and position information (coordinates) of a person frame on a camera image.

[0061] On the other hand, if the person detected from the current camera image is already being tracked (Yes in ST105), the face matching process in ST106 is skipped. In this case, the result of the face matching process (user ID) performed when the person tracking process was newly started is carried over.

[0062] Although the generated personal information includes a person ID, a user ID, and position information (coordinates) of a person's frame on a camera image, vital data (pulse, heart rate, stress, etc.) acquired from the person's image can also be included. Also, various types of health care data (body temperature, blood pressure, etc.) of the user may be acquired using a vital sensor other than a camera and managed as personal information of the user.

[0063] Next, a description will be given of the movement information acquisition process performed by the management server 2. Fig. 4 is a flowchart showing the procedure of the movement information acquisition process.

[0064] The management server 2 generates movement information representing the movement of each part of the person's body detected from the camera image (movement information generation process). This movement information generation process is executed every time the flow shown in Fig. 4 receives a camera image (frame) at each time and executes the person information acquisition process.

[0065] Here, while a person is receiving a service, some kind of movement of the person's body is observed. Therefore, by acquiring the period during which some kind of movement of the person's body is observed (movement period), it is possible to roughly grasp the period during which the person is receiving the service. As a result, when identifying the period during which the service is being provided based on the recognition results of the action recognition process, the processing load can be reduced and the processing accuracy can be improved by narrowing down the period to be processed to the movement period.

[0066] In this embodiment, first, a rectangular human frame area is cut out from a camera image based on the position information of the human frame acquired in the human information acquisition process, and a human image is acquired. Next, the positions of the human joints are estimated based on the human image, and position information (coordinates) of each joint is acquired as an estimation result (joint estimation process). In particular, whole-body joint estimation is performed here, which estimates the position of each joint for the entire body of the human. In this case, for example, approximately 17 joints (left and right wrists, elbows, shoulders, knees, ankles) and appropriate parts other than joints (head, eyes, ears, etc.) are set as estimation targets.

[0067] Joint estimation processing can be performed using an image recognition model (machine learning model) constructed by machine learning such as deep learning. An image of a person is input into the image recognition model, and the estimation results output from the image recognition model, i.e., the position information (coordinates) of each joint, are obtained.

[0068] 4, in the movement information acquisition process, first, the processor 13 acquires a camera image and person information acquired in the person information acquisition process (ST201). Note that the person information includes a person ID, a user ID, and position information (coordinates) of a person frame on the camera image.

[0069] Next, the processor 13 extracts the rectangular area of ​​the person frame from the camera image based on the position information of the person frame to acquire the person image (ST202).

[0070] Next, processor 13 detects the movement of each part of the person's body based on the person image (ST203). At this time, the positions of the person's joints are estimated based on the person image, and position information (coordinates) of the joints is obtained as the detection result (joint estimation process).

[0071] Next, the processor 13 generates and outputs movement information (ST204). The movement information includes the person ID and position information (coordinates) of each joint.

[0072] In this embodiment, the movement information generation process involves performing joint estimation on the entire body and acquiring position information for each joint of the entire body as the detection result, but the movement information generation process is not limited to this. However, depending on the content of the service received by the user, the user may only move part of their body, so it is preferable to be able to detect the movement of part of the body rather than the movement of the whole body.

[0073] For example, the movement information generation process may involve performing joint estimation on fingers and acquiring position information (coordinates) of each joint of the fingers as a detection result. In this case, position information of each joint of the fingers of the left and right hands may be acquired for a predetermined number of joints (including the tips of each finger), such as three joints of each finger (for example, 21 for one hand).

[0074] The movement information generation process may also be performed by detecting a person's hands, setting a rectangular frame (hand area) surrounding the hands, and acquiring position information (coordinates) of the hand area on the camera image as a detection result. Furthermore, it may also be performed by detecting a specific part of the person other than the hands (arms or legs), and acquiring position information of the target part on the camera image.

[0075] Next, a description will be given of the action recognition processing performed by the management server 2. Fig. 5 is an explanatory diagram showing an overview of the action recognition processing.

[0076] In the management server 2, a process of recognizing a person's movement (movement recognition process) is performed based on the camera image (frame) received from the camera 1.

[0077] At this time, first, a plurality of consecutive camera images (frames) are integrated to generate a clip video of a predetermined time (for example, 1 second or 2 seconds) (clip video generation process). In this embodiment, the camera images (frames) periodically transmitted from camera 1 at each time are integrated while shifting them one by one, so that a clip video of a predetermined time is generated each time a camera image is received at each time. In the example shown in FIG. 5, the clip video is made up of 30 camera images (frames). Here, if the frame rate is 30 fps, a clip video of 1 second is generated by integrating 30 camera images, and a clip video of 2 seconds is generated by integrating 60 camera images.

[0078] Furthermore, the action recognition process uses an image recognition model (machine learning model) for action recognition that is constructed by machine learning such as deep learning. Here, the image recognition model for action recognition is used to perform a process (classification process) of classifying the images into action classes corresponding to the services provided to users. Specifically, the clip video acquired in the clip video generation process is input to the image recognition model, and the image recognition model outputs an action class as the recognition result. Note that, for example, a 3DCNN (3D Convolutional Neural Network) may be adopted as the image recognition model for action recognition.

[0079] In addition, the action recognition process generates action class information for each time point for the target person. This action class information includes the position information of the person frame on the camera image, the name of the action class (class ID), and the recognition score of the action class.

[0080] Next, a description will be given of the action recognition processing performed by the management server 2. Fig. 6 is a flow diagram showing the steps of the action recognition processing.

[0081] The management server 2 performs a process (motion recognition process) to recognize the motion of the target person based on the camera image received from the camera 1. This motion recognition process is executed every time the flow shown in Fig. 6 receives a camera image (frame) periodically transmitted from the camera 1 at each time.

[0082] First, the processor 13 acquires a camera image (frame) received from the camera 1 via the communication unit 11 (image acquisition process) (ST301).

[0083] Next, processor 13 determines whether the number of camera images acquired at each time point has reached a predetermined number (ST302). Note that once the number of camera images has reached the predetermined number, all subsequent camera images are determined to have reached the predetermined number.

[0084] Here, when the number of acquired camera images reaches a predetermined number (for example, 30) (Yes in ST302), processor 13 then generates a clip video by integrating the predetermined number of camera images (ST303). At this time, the predetermined number of most recent camera images including the current camera image are integrated. Therefore, the current clip video will be one camera image shifted from the previous clip video.

[0085] Next, processor 13 inputs the clip video into an image recognition model for action recognition and obtains an action class output from the image recognition model (ST304). This action class represents the type of service (functional training, meal, etc.) to be provided to the user.

[0086] Next, the processor 13 outputs action class information as a recognition result of the action recognition process (ST305). This action class information includes position information of the person frame on the camera image, the name of the action class (class ID), and the recognition score of the action class.

[0087] Next, an overview of the integrated determination process performed by the management server 2 will be described. Fig. 7 is an explanatory diagram showing an overview of the integrated determination process.

[0088] The management server 2 performs a process (integrated judgment process) to identify the services provided to the user by combining the movement information obtained in the movement information generation process and the movement class information obtained in the movement recognition process to make an integrated judgment.

[0089] In the integrated determination process, first, a period during which some kind of movement is occurring in the person's body (movement period) is acquired based on the movement information acquired in the movement information generation process (movement period acquisition process).

[0090] In the movement period acquisition process, first, the amount of change (e.g., movement distance) of each joint is calculated based on the position information (coordinates) of each joint. Next, the amount of change of each joint is compared with a predetermined threshold, and joints with movement are detected based on whether the amount of change of each joint is equal to or greater than the threshold. Next, the number of joints with movement is compared with a predetermined threshold (e.g., half the total number of joints), and it is determined whether the number of joints with movement is equal to or greater than the threshold. Then, if the number of joints with movement is equal to or greater than the threshold, it is determined that some movement is occurring in the target person's body.

[0091] In the integrated determination process, next, the movement class information of each time included in the movement period is extracted from the movement class information of the target person stored in the storage unit 12 (movement class information extraction process).

[0092] In the integrated judgment process, the type of service provided to the target person is then identified based on the extracted action class information, i.e., the action class information for each time included in the movement period, and the period during which the service is provided (start time and end time) is identified (service identification process).

[0093] In the service identification process, the recognition score of the action class included in the action class information at each time is compared with a predetermined threshold, and if the recognition score is equal to or greater than the threshold, it is determined that the service corresponding to the action class is being provided.Then, the range in which the action class is the same and the determination results of the recognition score being equal to or greater than the threshold are consecutive is set as the service implementation period.In other words, the time when the recognition score first reaches or exceeds the threshold is set as the start time, and the time when the recognition score last reaches or exceeds the threshold is set as the end time.In addition, the type of service provided to the target person is identified based on the action class common to the action class information at each time included in the service implementation period.

[0094] Finally, in the integrated judgment process, service implementation information is generated as the judgment result and stored in the storage unit 12. The service implementation information includes the user ID, the service ID, and information related to the period of service implementation (start time and end time). The service ID is information that identifies the type of service (functional training, meal, etc.) provided to the user.

[0095] In this embodiment, the joints of the target person are estimated from the camera image, and whether or not there is any movement in the target person's body is determined based on the movement of the joints. However, the target person's hands may be detected from the camera image, and whether or not there is any movement in the target person's body may be determined based on the movement of the hands. In this case, the amount of change (e.g., movement distance) of a rectangular hand area set on the camera image may be obtained based on the hand detection result, and it may be determined whether the amount of change in the hand area is equal to or greater than a threshold.

[0096] Next, a description will be given of the procedure of the integrated determination process performed by the management server 2. Fig. 8 is a flow diagram showing the procedure of the integrated determination process.

[0097] In the management server 2, the point at which the service provided to the user has ended (service end point) is used as a trigger to perform an integrated determination process (movement period acquisition process, movement class information extraction process, and service identification process). This integrated determination process is performed each time the flow shown in Fig. 8 receives camera images (frames) at each time and executes person information acquisition process, movement information generation process, and movement recognition process.

[0098] Here, when a new service is started for a user, the person moves from another location to the location where the new service is provided immediately before that. Also, when a service that has been provided to the person ends, the person moves to the location where a different service is provided immediately after that. Therefore, in this embodiment, when the movement of a person is detected, it is determined whether the current time is the service start point or the service end point, and if the current time is the service end point, an integrated determination process is started.

[0099] In addition, the movement of a person that triggers the start of the integrated judgment process includes not only the movement of a person within the shooting area of ​​camera 1, but also when a person leaves the shooting area of ​​camera 1 to move to another area and disappears from the camera image.

[0100] In the integrated judgment process, as shown in Fig. 8, first, processor 13 acquires person information acquired in person information acquisition process, motion information acquired in motion information acquisition process, and motion class information acquired in motion recognition process (ST401). The person information includes a person ID, a user ID, and position information (coordinates) of a person frame on a camera image. The motion information includes a person ID and position information (coordinates) of each joint. The motion class information includes position information of a person frame on a camera image, a name of a motion class (class ID), and a recognition score of the motion class.

[0101] Next, processor 13 detects the movement of the person based on the position information of the person frame (ST402). At this time, the position of the person frame detected in the current camera image is compared with the position of the person frame detected in the previous camera image, and if the amount of change in the person frame (movement distance) is equal to or greater than a predetermined threshold, it is determined that the person has moved.

[0102] Next, processor 13 determines the service start point and the service end point (ST403). Here, the service start point represents the time when a new service is started for the user, and the service end point represents the time when the service provided to the user up until now has ended. At this time, if a person's movement is detected for the first time, the current time is set as the service start point. Also, if the service start point was set when a person's movement was most recently detected, the current time is set as the service end point. Also, if the service end point was set when a person's movement was most recently detected, the current time is set as the service start point.

[0103] Next, the processor 13 determines whether or not the current time corresponds to the service end point (ST404).

[0104] If this does not correspond to the service end point (No in ST404), the process returns to ST401 and proceeds to the processing for the next time.

[0105] On the other hand, if the current point corresponds to the end of the service, i.e., if the service provided to the user up until now has ended (Yes in ST404), then processor 13 estimates the period during which some movement is occurring in the person's body (movement period) based on the movement information (movement period acquisition process) (ST405).

[0106] Next, the processor 13 extracts the movement class information of each time included in the movement period from the movement class information of the target person stored in the storage unit 12 (movement class information extraction process) (ST406).

[0107] Next, processor 13 identifies the type of service provided to the target person based on the extracted action class information, and also identifies the period (start time and end time) during which the service is provided (service identification process) (ST407). At this time, the recognition score of the action class included in the action class information is compared with a predetermined threshold, and if the recognition score is equal to or greater than the threshold, it is determined that the service corresponding to the action class is being provided.

[0108] Next, the processor 13 generates and outputs service execution information (ST408). This service execution information includes the user ID, the service ID, and information relating to the execution period (start time and end time) of the service.

[0109] Next, a description will be given of the service implementation information managed by the management server 2. Fig. 9 is an explanatory diagram showing the contents of the service implementation information.

[0110] In the management server 2, service implementation information regarding the implementation status of the service provided to the user is generated for each user through a person information acquisition process, a movement information acquisition process, an action recognition process, an integrated determination process, etc. This service implementation information for each user is stored in the storage unit 12. The service implementation information includes a user ID (identification information of the user), a service ID (identification information of the service), information regarding the implementation period of the service (start time and end time), and the file name of the camera image (identification information of the camera image).

[0111] Next, a description will be given of the service implementation status confirmation screen 21 displayed on the management terminal 3. FIG.

[0112] The service implementation status confirmation screen 21 has an information display field 22 for each service provided to the user. The implementation period and planned period of the corresponding service are displayed in the information display field 22. By visually checking the implementation period and planned period in the information display field 22, the care facility manager can immediately grasp the status of the discrepancy between the implementation period and the planned period.

[0113] At this time, the management server 2 reads out the service plan information and service implementation information of the relevant user from the memory unit 12, obtains the planning period contained in the service plan information and the implementation period contained in the service implementation information, generates display information for the service implementation status confirmation screen 21, and distributes the display information to the management terminal 3.

[0114] In this embodiment, the service plan information and the service implementation information are output in association with each other, so that the administrator can easily check whether the provision of services to users is being carried out as planned.

[0115] Furthermore, the information display column 22 for a service for which there is a significant discrepancy between the implementation period and the planned period is highlighted on the service implementation status confirmation screen 21. In the example shown in Fig. 10, the discrepancy between the implementation period and the planned period is significant for the "meal" service, and the information display column 22 for "meal" is highlighted in a color different from the other information display columns 22. This allows the administrator to easily grasp the services for which there is a significant discrepancy between the implementation period and the planned period.

[0116] Furthermore, the IoU (Intersection over Union) for each service is displayed in the information display field 22 for each service. IoU is an index that indicates the degree to which the implementation period and the planning period overlap, and is calculated using the following formula. IoU = (overlapping period between implementation and planning periods) / (integrated period between implementation and planning periods)

[0117] At this time, the management server 2 calculates the IoU for each service based on the service plan information and service implementation information. Next, the IoU for each service is compared with a predetermined threshold, and if the IoU is smaller than the threshold, it is determined that there is a significant discrepancy between the implementation period and the planned period. Next, if there is a service that is determined to have a significant discrepancy between the implementation period and the planned period, display information for the service implementation status confirmation screen 21 is generated in which the information display field 22 for that service is highlighted, and the display information is delivered to the management terminal 3.

[0118] In this embodiment, to notify the administrator that there is a service with a significant discrepancy between the implementation period and the planned period, the information display field 22 of the relevant service is highlighted, but this is not limiting. For example, an alert screen may be displayed or an alert sound may be output.

[0119] Furthermore, a camera image 23 is displayed on the service implementation status confirmation screen 21. When the administrator operates the information display field 22 for a service of interest, the camera image 23 corresponding to that service is displayed. This allows the administrator to easily check the actual status of the service being provided to the user.

[0120] The service implementation status confirmation screen 21 also has a playback operation section 24. By operating the playback operation section 24, the administrator can play back the camera images 23 as moving images. This allows the administrator to view camera images at any time.

[0121] Here, the service implementation information (see Figure 9) managed by the management server 2 includes the file name of the camera image as link information for displaying on the management terminal 3 the camera image 23 corresponding to the person and service selected by the administrator.

[0122] The management server 2 reads out from the memory unit 12 the service implementation information corresponding to the person and service selected by the administrator, and based on the file name of the camera image included in the service implementation information, reads out from the memory unit 12 a camera image showing the implementation status of the service for the target user, and delivers the camera image to the management terminal 3.

[0123] The camera image files correspond to the implementation period of the service. Furthermore, the service implementation information may include a camera ID as identification information for the camera image instead of the file name of the camera image. In this case, the camera images corresponding to the implementation period of the service can be extracted based on the implementation period (start time and end time) included in the service implementation information.

[0124] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0125] The nursing care information recording device and nursing care information recording method of the present invention have the effect of automating the recording of the implementation status of services provided to users in nursing care facilities, thereby reducing the burden on staff, and are useful as nursing care information recording devices and nursing care information recording methods that record and manage the implementation status of services provided to users in nursing care facilities. [Explanation of symbols]

[0126] 1 camera 2. Management server (care information recording device, information processing device) 3. Management terminal (administrator device) 11 Communications Department 12 Storage section 13 processors 21 Service implementation status confirmation screen 22 Information display column 23 camera images 24 Playback operation section

Claims

1. A care information recording device that uses a processor to execute a process of recording and managing the implementation status of services provided to users in a care facility, The processor: Identifying the target person as a user based on a camera image of the target person and acquiring user identification information; Based on the camera image, a movement period associated with the provision of the service to the target person is acquired based on an amount of change in movement information representing a movement of a predetermined part of the target person's body; sequentially inputting a clip video consisting of a plurality of consecutive camera images into an image recognition model to sequentially acquire action class information; Identifying the type of service provided to the target person and its implementation period based on the movement period and the service identification information and recognition score included in the action class information; A care information recording device that generates service implementation information including the user identification information and information regarding the type of service and its implementation period.

2. The processor: The care information recording device according to claim 1, wherein the target person is identified by face matching between the target person detected from the camera image and a pre-registered user.

3. The processor:

2. The care information recording device according to claim 1, wherein the positions of the joints of the subject person are estimated based on the camera image, and the movement information including joint position information as the estimation result is acquired.

4. The processor:

2. The care information recording device according to claim 1, further comprising: a storage unit that reads out service plan information relating to a schedule for providing a service to a user; and outputs the service plan information and the service implementation information in association with each other.

5. The processor: The nursing care information recording device described in claim 4, characterized in that the implementation period for each type of service included in the service implementation information is matched with the planned period for each type of service included in the service plan information, the overlapping period and the integrated period are determined in the correspondence between the implementation period and the planned period for each type of service, an index representing the ratio of the overlapping period to the integrated period is compared with a predetermined threshold value for each type of service, and service types for which the index is smaller than the threshold value are identifiable.

6. The processor: The nursing care information recording device described in claim 1, characterized in that the file name of the camera image is added to the service implementation information as link information for displaying the camera image corresponding to the person and type of service selected by the administrator on the administrator device.

7. A care information recording method that causes an information processing device to perform a process of recording and managing the implementation status of services provided to users in a care facility, The information processing device includes: Identifying the target person as a user based on a camera image of the target person and acquiring user identification information; Based on the camera image, a movement period associated with the provision of the service to the target person is acquired based on an amount of change in movement information representing a movement of a predetermined part of the target person's body; sequentially inputting a clip video consisting of a plurality of consecutive camera images into an image recognition model to sequentially acquire action class information; Identifying the type of service provided to the target person and its implementation period based on the movement period and the service identification information and recognition score included in the action class information; A care information recording method characterized by generating service implementation information including the user identification information and information regarding the type of service and its implementation period.

Images