Caregiving video analysis system, method for analyzing caregiving video, and caregiving video analysis program

Abstract

To provide a caregiving video analysis system, a method for analyzing a caregiving video, and a caregiving video analysis program which are applicable.SOLUTION: A caregiving video analysis system (1) includes an acquisition unit (211), a posture detection unit (213), a state determination unit (214), and a notification unit (215). The acquisition unit (211) acquires audio in a room where a person is present and an image including the person. The posture detection unit (213) detects the posture of the person included in the image. The state determination unit (214) determines the state of the person on the basis of audio related to the person acquired by the acquisition unit (211) and the posture of the person detected by the posture detection unit (213). The notification unit (215) notifies a determination result made by the state determination unit (214).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a care video analysis system, a care video analysis method, and a care video analysis program. [Background technology]

[0002] Japan has seen a remarkable increase in life expectancy due to improvements in living standards, sanitary conditions, and medical standards that came with the rapid economic growth following the war. This, coupled with a declining birthrate, has led to an aging society with a high aging rate. In such an aging society, an increase in people requiring nursing care or care due to illness, injury, aging, etc. (hereinafter referred to as "care recipients") is expected. In nursing care facilities such as hospitals and elderly welfare facilities (hereinafter referred to simply as "facilities"), care is provided to care recipients by caregivers and nurses (hereinafter referred to as "care staff").

[0003] When an incident occurs at a facility that threatens the safety and security of a care recipient, such as a fall, aspiration, or slipping off a bed, care staff must rush to the recipient's side immediately and provide first aid, etc. Even if immediate action is not required, there may be cases where measures are necessary to prevent the accident from recurring or to improve the living environment.

[0004] Furthermore, care for a care recipient may be provided by multiple care staff, but in many cases, a single care staff member is in charge of providing care in the care recipient's room, etc. When a single care staff member provides care for a care recipient in a space where it is difficult for others to see what is happening inside the room, such as the care recipient's room, there have been cases where the care recipient has suffered violence or abuse from the care staff, which has become a problem.

[0005] In relation to this, Patent Document 1 below discloses a video recording system that is configured to constantly record the care site, but when a problem occurs between the caregiver and the person being cared for, only a specified period of video footage before and after the problem occurs is recorded. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-67508 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the video recording system described in Patent Document 1 is intended to detect trouble between the caregiver and the care recipient, and is not intended to detect abnormalities in the care recipient's condition, such as falls, aspiration, or slipping off the bed, when the care recipient is alone in the room. Therefore, a caregiver in another room or a manager managing the caregiver cannot notice abnormalities in the care recipient's condition when the care recipient is alone in the room. Furthermore, in the case of a care recipient slipping or falling off the bed, the care recipient may be close to or overlapping the floor, bed, or comforter, and therefore it may not be possible to accurately determine whether the care recipient's condition is abnormal based solely on the video footage of the room.

[0008] The present invention has been made in consideration of the above circumstances, and its main purpose is to provide a care video analysis system, a care video analysis method, and a care video analysis program that can accurately determine abnormalities in the condition of a care recipient. [Means for solving the problem]

[0009] The above-mentioned problems of the present invention are solved by the following means.

[0010] (1) A nursing care video analysis system having an acquisition unit that acquires audio from within a room where a person is present and an image including the person, a posture detection unit that detects the posture of the person included in the image, a status determination unit that determines the state of the person based on the audio related to the person acquired by the acquisition unit and the posture of the person detected by the posture detection unit, and a notification unit that notifies the result of the determination made by the status determination unit.

[0011] (2) The nursing video analysis system described in (1) above, wherein the condition determination unit determines that the person's condition is abnormal when the person's posture is maintained the same for a certain period of time and the audio acquired by the acquisition unit is a specific audio related to the person.

[0012] (3) A nursing video analysis system as described in (2) above, further comprising a voice recognition unit that converts voice related to the person into text, and the state determination unit determines that the voice is a specific voice related to the person if the text converted by the voice recognition unit includes specific words that ask others for help.

[0013] (4) A nursing video analysis system as described in (1) or (2) above, wherein the same posture is one of a sitting posture, a lying posture, a standing posture, and a posture leaning against an object.

[0014] (5) The nursing video analysis system described in (1) or (2) above, wherein the posture detection unit estimates the person's skeleton and detects the person's posture based on the estimated skeleton.

[0015] (6) A nursing video analysis system as described in (1) or (2) above, further comprising a sound collection unit installed in the room to collect sound within the room, and a photographing unit installed in the room to photograph the person in the room, wherein the acquisition unit acquires the sound collected by the sound collection unit and acquires the image photographed by the photographing unit.

[0016] (7) The nursing video analysis system described in (6) above, wherein the sound collection unit and the imaging unit are housed in a sensor box installed on the ceiling of the room.

[0017] (8) A nursing video analysis system as described in (1) or (2) above, wherein the notification unit notifies the judgment result to a terminal of a care staff member providing care to the person and / or a terminal of a manager of the care staff member.

[0018] (9) A nursing video analysis system as described in (2) above, wherein the specific sounds include human voices produced due to pain, surprise, or fright, or sounds of contact between people or between people and objects.

[0019] (10) The nursing video analysis system described in (6) above, further comprising a recording unit that records the image acquired by the acquisition unit when the condition determination unit determines that the person's condition is abnormal.

[0020] (11) The nursing video analysis system described in (6) above, wherein the acquisition unit acquires the audio collected by the audio collection unit when the level of the audio collected by the audio collection unit is above a predetermined threshold.

[0021] (12) A method for analyzing nursing care video, comprising the steps of: (a) acquiring audio from within a room where a person is present and an image including the person; (b) detecting the posture of the person included in the image; (c) determining the state of the person based on the audio related to the person acquired in step (a) and the posture of the person detected in step (b); and (d) notifying the result of the determination in step (c).

[0022] (13) The nursing video analysis method described in (12) above, further comprising a step (e) of recording the image acquired in step (a) if the condition of the person is determined to be abnormal in step (c).

[0023] (14) A care video analysis program for causing a computer to execute the processing included in the care video analysis method described in (12) or (13) above. [Effects of the Invention]

[0024] According to the present invention, the condition of the care recipient is determined based on the voice related to the care recipient and the posture of the care recipient, so that abnormalities in the condition of the care recipient can be determined with high accuracy. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a monitoring system according to an embodiment of the present invention. [Figure 2] 2 is a diagram showing the interior of a room in which a care recipient is staying, in which the detection unit shown in FIG. 1 is arranged. FIG. [Figure 3] 2 is a block diagram illustrating a hardware configuration of a detection unit shown in FIG. 1. FIG. [Figure 4] 2 is a block diagram illustrating an example of a hardware configuration of the server shown in FIG. 1. [Figure 5] 5 is a schematic block diagram illustrating the main functions of a control unit of the server shown in FIG. 4. [Figure 6] 2 is a block diagram illustrating a hardware configuration of the fixed terminal shown in FIG. 1. [Figure 7] FIG. 2 is a block diagram showing a hardware configuration of the mobile terminal shown in FIG. [Figure 8] 10 is a flowchart illustrating a processing procedure of a care video analysis method performed by the care video analysis device. [Figure 9] FIG. 10 is a schematic diagram showing an example of a captured image of the interior of a care recipient's room. [Figure 10] 10A and 10B are schematic diagrams illustrating examples of joint point estimation results for a person included in a photographed image. [Figure 11] FIG. 10 is a schematic diagram showing an example of an image captured inside a room when a care recipient falls. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, a nursing care video analysis system, a nursing care video analysis method, and a nursing care video analysis program according to an embodiment of the present invention will be described with reference to the drawings. Note that in the drawings, identical elements are designated by the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0027] (Embodiment) <Monitoring System 1> 1 is a diagram showing the overall configuration of a monitoring system 1 according to one embodiment. The monitoring system 1 of this embodiment also functions as a care video analysis system. The care video analysis system of this embodiment is a system that detects danger or trouble that occurs to a care recipient 80 who is staying at a facility based on the voice and video of the care recipient 80, notifies facility staff such as care staff 70, and keeps video related to the danger or trouble as evidence.

[0028] The monitoring system 1 includes a plurality of detectors 10, a server 20, a fixed terminal 30, and one or more mobile terminals 40. These are connected to each other via a network 50, such as a local area network (LAN), a telephone network, or a data communication network, either wired or wirelessly, so that they can communicate with each other. The network 50 may include a repeater that relays communication signals. In the example shown in FIG. 1, the detectors 10, the server 20, the fixed terminals 30, and the mobile terminals 40 are connected to each other so that they can communicate with each other via a network 50, such as a wireless LAN (for example, a LAN conforming to the IEEE 802.11 standard) that includes an access point 51.

[0029] The monitoring system 1 is suitably installed in buildings such as hospitals and elderly welfare facilities (nursing care facilities). In the example shown in Fig. 1, the monitoring system 1 is installed in a building of a facility that has a plurality of rooms including a plurality of rooms and a care station in which a plurality of care recipients 80 each reside. Hereinafter, the rooms in which the care recipients 80 each reside will also be simply referred to as "rooms."

[0030] The detection units 10 are arranged in the rooms of each floor (floors (1) to (3)), which are the observation areas for the care recipients 80. In the example shown in FIG. 1, on floor (1), four detection units 10 are arranged in the rooms of care recipients 80 A, B, C, and D. Although not shown, a plurality of detection units 10 are also arranged in the rooms of care recipients 80 on other floors. Each care staff member 70 carries a mobile terminal 40. The server 20 may be an external server connected to the network 50. Furthermore, the fixed terminal 30 may be omitted, and the server 20 or the mobile terminal 40 may have its functions.

[0031] <Detection unit 10> Fig. 2 is a diagram showing the interior of a room of a care recipient 80 in which the detection unit 10 shown in Fig. 1 is arranged. Fig. 3 is a block diagram showing an example of the hardware configuration of the detection unit 10 shown in Fig. 1.

[0032] 2, a sensor box having a camera 14 and a microphone 15 is disposed, for example, on the ceiling of a room. The camera 14 and the microphone 15 may be disposed on the upper part of a wall or attached to a bed 90. The care call unit 16 is disposed, for example, at an appropriate position on the wall.

[0033] 3, the detection unit 10 includes a control unit 11, a communication unit 12, a storage unit 13, a camera 14, a microphone 15, and a care call unit 16, and these components are connected to each other via a bus. Each component may be mounted in a single housing or in a separate housing.

[0034] The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The control unit 11 controls each part of the detection unit 10 and performs calculation processing according to a program.

[0035] The communication unit 12 is an interface circuit (for example, a LAN card, a wireless communication circuit, etc.) for communicating with, for example, the server 20 via a LAN.

[0036] The storage unit 13 includes a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 13 stores various programs and various data. In this embodiment, as will be described later, actions such as "getting up," "getting out of bed," and "falling" are detected as events, and images captured by the camera 14 within one minute before and after the occurrence of the event can be stored in the storage unit 13.

[0037] Camera 14 functions as a photographing unit and can photograph a photographing area including, for example, bed 90 and output the photographed image (image data). Camera 14 can be configured to include an imaging optical system and a two-dimensional imaging element. The photographed image includes a still image and a video. Camera 14 can photograph the photographing area as a video consisting of a plurality of photographed images (frames) at a frame rate of, for example, 15 fps to 30 fps. Camera 14 can be, for example, a visible light camera or a near-infrared camera. Hereinafter, the photographed image photographed by camera 14 will also be simply referred to as a "photographed image." The start and end of photographing by camera 14 is controlled by control unit 11.

[0038] The microphone 15 functions as a sound collector and collects sounds from within the room of the care recipient 80. The microphone 15 can be housed in a sensor box installed on the ceiling of the room together with the camera 14. The start and end of sound collection by the microphone 15 is controlled by the control unit 11.

[0039] The care call unit 16 includes a push-button switch and detects a care call when the care recipient 80 presses the switch. Instead of a push-button switch, a care call may be detected based on the voice of the care recipient 80 using an audio microphone. The care call unit 16 may also include a microphone and speaker for voice communication with the mobile terminal 40.

[0040] The control unit 11 recognizes the behavior of the care recipient 80 from the captured images. The recognized behaviors include, for example, getting up, getting out of bed, getting out of a seat, falling, slipping, walking, and going out. For example, the control unit 11 detects an image silhouette (hereinafter referred to as a "human silhouette") from multiple captured images (e.g., video images). The human silhouette can be detected, for example, by extracting a range of pixels where the time difference between images captured at different times is relatively large. The human silhouette may be detected from the captured images using a background subtraction method. The human silhouette may be replaced by joint points detected from the captured images using a trained neural network model. The behavior of the care recipient 80 can be detected from temporal changes in the posture (e.g., standing, sitting, and lying down) of the care recipient 80 recognized based on the human silhouette. The behavior of the care recipient 80 may also be detected from the relative positional relationship between the human silhouette and objects in the room, such as the bed 90. For example, "getting up" may be detected as a behavior (state) in which a human silhouette crosses the area of ​​the bed 90 that is set in advance as a coordinate area in the captured image.

[0041] Information about the behavior of the care recipient 80 detected by the detection unit 10 can be reflected in the care record of the care recipient 80.

[0042] <Server 20> FIG. 4 is a block diagram illustrating an example of the hardware configuration of server 20. Server 20 includes a control unit 21, a communication unit 22, and a storage unit 23, which are interconnected by a bus. Server 20 functions as a video analysis device. FIG. 5 is a block diagram illustrating an example of an outline of the main functions of control unit 21 of server 20 shown in FIG. 4.

[0043] The control unit 21 includes a CPU, RAM, ROM, etc. (not shown). The control unit 21 controls each unit of the server 20 and performs calculation processing according to a nursing video analysis program. As shown in FIG. 5, the control unit 21 functions as an acquisition unit 211, a voice recognition unit 212, a posture detection unit 213, a state determination unit 214, and a notification unit 215. An overview of each function is as follows:

[0044] The acquisition unit 211 acquires audio from within a room where a person (e.g., the care recipient 80) is present and a captured image including the care recipient 80. The voice recognition unit 212 converts audio related to the care recipient 80 into text. The posture detection unit 213 detects the posture of the care recipient 80 included in the captured image. In this specification, posture may include, for example, a sitting position, a lying position, an upright position, and a posture leaning against an object. The state determination unit 214 determines the state of the care recipient 80 based on the audio related to the care recipient 80 acquired by the acquisition unit 211 and the posture of the care recipient 80 detected by the posture detection unit 213. In this specification, the state of a person may be, for example, a state in which the person is calling for help after a fall or tumble, a state in which the person has slipped off the bed, a state in which the person is suffering from pain, coughing, aspiration or ingestion, vomiting, or the like, a state in which the person is frightened, or the like. The notification unit 215 notifies the result of the determination made by the state determination unit 214 .

[0045] The communication unit 22 is configured with an interface circuit for communicating with, for example, the detection unit 10, the fixed terminal 30, the mobile terminal 40, and the like via a LAN.

[0046] The storage unit 23 is configured with an HDD, an SSD, etc., and stores an OS (Operating System), various programs such as a care video analysis program, and various data. The various data include, for example, care records for each care recipient 80, authentication information for the care staff 70 and the care recipient 80, etc. In this embodiment, the storage unit 23 also stores voices collected by the microphone 15, captured images, posture detection results detected by the posture detection unit 213, and state determination results determined by the state determination unit 214.

[0047] The recording unit 24 records the images captured by the camera 14. The recording unit 24 has a sufficient capacity to record the images captured by the camera 14 for a predetermined period of time. The recording unit 24 may be provided outside the care video analysis device. The predetermined period may be about one day to several days, depending on the period at which the manager (hereinafter simply referred to as the "manager") who manages the care staff 70 checks the determination result by the state determination unit 214.

[0048] <Fixed terminal 30> 6 is a block diagram illustrating an example of the hardware configuration of fixed terminal 30. Fixed terminal 30 is a PC (Personal Computer) and includes a control unit 31, a communication unit 32, a storage unit 33, a display unit 34, and an input unit 35. The basic configurations of control unit 31, communication unit 32, and storage unit 33 are similar to the configurations of the corresponding elements of server 20, so redundant explanations will be omitted.

[0049] The display unit 34 is configured with a display (e.g., a liquid crystal display) and displays various information. The display unit 34 can display, for example, the determination result of the condition of the care recipient 80 determined by the condition determination unit 214. The input unit 35 is configured with a keyboard or a touch panel and accepts operational input of various information by a manager or the like.

[0050] The control unit 31 can accept input from the input unit 35 of a table in which the correspondence between a unique ID that identifies the detection unit 10 and the care recipient 80 is registered, transmit the table to the server 20, and store it in the memory unit 23.

[0051] Furthermore, the control unit 31 can receive, via the input unit 35, a table in which the correspondence between the unique ID that identifies the mobile terminal 40 and the care staff 70 is registered, and transmit it to the server 20, which can then store it in the storage unit 23.

[0052] In addition, the control unit 31 can receive a table via the input unit 35 that registers the correspondence between care staff 70 and the care recipients 80 that the care staff 70 are responsible for, send it to the server 20, and store it in the memory unit 23.

[0053] Furthermore, when the care staff 70 or technical staff installs the detection unit 10 in each room (residential room), the care staff 70 or technical staff or the like associates the room number with the detection unit 10 via the fixed terminal 30. The care staff 70 or technical staff or the like also calibrates and specifies the position information of objects installed in the room, such as the bed 90, i.e., the contour information of the upward view captured by the ceiling camera 14. The care staff 70 or technical staff or the like also associates the personal IDs (name, identification number, etc.) of the care recipients 80 residing in the room with each room number. The fixed terminal 30 manages a list of care recipients, including the personal IDs, room numbers, names of staff members in charge, and units to which the care recipients 80 belong. The list of care recipients can be shared within the monitoring system 1.

[0054] <Mobile terminal 40> 7 is a block diagram showing the hardware configuration of the mobile terminal 40. The mobile terminal 40 includes a control unit 41, a wireless communication unit 42, a storage unit 43, an input / display unit 44, an audio input / output unit 45, and a position detection unit 46, which are interconnected by a bus. The mobile terminal 40 may be configured by a portable communication terminal device such as a tablet computer, a smartphone, or a mobile phone. The basic configuration of the control unit 41 is similar to the configuration of the control unit 21 of the server 20, and therefore a redundant description will be omitted. The functions of the control unit 41 will be described in detail below.

[0055] The wireless communication unit 42 communicates wirelessly with each device directly or via an access point 61 using wireless communication standards such as Wi-Fi and Bluetooth (registered trademark).

[0056] The storage unit 43 is configured with an SSD or an SD card, and stores mobile applications compatible with the monitoring system 1, various programs, and various data.

[0057] The input display unit 44 is configured by, for example, a touch panel in which a touch sensor is superimposed on a display surface of a liquid crystal display, etc. The input display unit 44 displays various information and receives various inputs.

[0058] The audio input / output unit 45 is, for example, a speaker and a microphone, and enables the care staff 70 to make audio calls to other mobile terminals 40 via the wireless communication unit 42. The audio input / output unit 45 enables the care staff 70 to make audio calls to the care recipient 80 via the wireless communication unit 42 and the detection unit 10 via the server 20.

[0059] The position detection unit 46 is a GPS module or the like, and detects the position of the mobile terminal 40. As will be described later, the position of the mobile terminal 40 may be detected by the control unit 41 using signals transmitted from beacons installed at predetermined installation positions.

[0060] <Functions of the control unit 21> Fig. 8 is a flowchart for explaining the processing steps of a care video analysis method using a care video analysis device. The processing of the flowchart shown in the figure is realized by the CPU of the control unit 21 executing a care video analysis program. Fig. 9 is a schematic diagram showing an example of a captured image of the inside of the care recipient's room. Fig. 10 is a schematic diagram illustrating an example of an estimation result of joint points for a person included in the captured image. Fig. 11 is a schematic diagram showing an example of a captured image of the inside of the care recipient's room when the care recipient falls.

[0061] [When photographing and collecting sound from inside the room of care recipient 80, and analyzing the video (photographed images) in real time] Assume that the care recipient 80, who is sitting on a bed 90 in his / her room, gets off the bed 90 and starts walking. The behavior of the care recipient 80 is captured by the camera 14 of the detection unit 10 installed on the ceiling. As described above, the camera 14 captures an image of an image capturing area including, for example, the bed 90, and outputs the captured image. The image capturing area includes at least the care recipient 80. Furthermore, voices and sounds made by the care recipient 80 are collected by the microphone 15 of the detection unit 10.

[0062] As shown in FIGS. 8 and 9 , the acquisition unit 211 acquires a sound from within a room where a person 500 is present and a captured image 601 including the person 500 (step S101). The person 500 included in the captured image 601 is assumed to be an object corresponding to the care recipient 80. Identification of the person 500 will be described later. The bed 700 is an object corresponding to the bed 90. The microphone 15 of the detection unit 10 collects a sound from within the room. The acquisition unit 211 acquires the sound collected by the microphone 15. The acquisition unit may be configured to acquire the sound collected by the microphone 15 when the level of the sound collected by the microphone 15 is equal to or higher than a predetermined threshold. This configuration can prevent unwanted noise, such as normal sounds or talking, from being mixed into the sound when determining the condition of the care recipient 80. The predetermined threshold may be, for example, a level corresponding to a sound with a large amplitude, such as a scream or a groan. The predetermined threshold may be set experimentally or empirically by, for example, the care staff member 70 or a manager.

[0063] The acquisition unit 211 also acquires the captured image captured by the camera 14. In FIG.

[0064] Next, the posture detection unit 213 detects the posture of the care recipient 80 based on the captured image 601 (step S102). As shown in FIG. 10, the posture detection unit 213 estimates joint points of the person 500 included in the captured image 601 and detects the skeleton of the person 500. In the same figure, estimated joint points 510 are indicated by "circles." The joint points 510 estimated by the posture detection unit 213 may include, for example, eyes (left and right), ears (left and right), nose, elbows (left and right), wrists (left and right), shoulders (left and right), neck, waist (left and right), pelvis, knees (left and right), and ankles (left and right). Furthermore, the skeleton of the person 500 may be represented by line segments connecting the joint points 510.

[0065] For example, the posture detection unit 213 can detect the human rectangle 501 using a trained model of a neural network that has been trained to estimate a human rectangle 501 ( FIG. 9 ) including the person 500 from the captured image 601. The posture detection unit 213 can then estimate the joint points 510 of the person 500 using a trained model of a neural network that has been trained to estimate the joint points 510 from the human rectangle 501. An example of a trained model for estimating the human rectangle 501 from the captured image 601 is an RPN (Region Proposal Network) model. Also, an example of a trained model for detecting the person's joint points 510 from the human rectangle 501 is a Deep Pose, a CNN (Convolution Neural Network), a Res Net, or the like. In the example shown in FIG. 9 , the posture detection unit 213 determines that the care recipient 80 is sitting, i.e., the posture of the care recipient 80 is a sitting position.

[0066] Next, the state determination unit 214 determines the state of the care recipient 80 (step S103). For example, the state determination unit 214 determines the state of the care recipient 80 based on the voice related to the care recipient 80 acquired by the acquisition unit 211 and the posture of the care recipient 80 detected by the posture detection unit 213. More specifically, the state determination unit 214 determines that the state of the care recipient 80 is abnormal if the posture of the care recipient 80 is maintained the same for a certain period of time and the voice acquired by the acquisition unit 211 is a specific voice related to the care recipient 80. The certain period of time may be, for example, several seconds to several minutes. On the other hand, if the posture of the care recipient 80 is not maintained the same for the certain period of time, the state determination unit 214 determines that the state of the care recipient 80 is normal. Furthermore, even if the posture of the care recipient 80 is maintained the same for a certain period of time, if the voice acquired by the acquisition unit 211 is not a specific voice related to the care recipient 80, the condition of the care recipient 80 is determined to be normal.

[0067] The specific sounds may include, for example, sounds made when a person is in an abnormal state or feels abnormal, such as sounds made due to pain or fear. The specific sounds may also include sounds of contact between people or between people and objects. More specifically, sounds made due to pain or fear include calls for help from others, such as "Ouch," "Help me," "Uh," "Waah," and "Ahhh," as well as groans and screams. Sounds made when coughing or vomiting may also be included in sounds made due to pain or fear. Sounds of contact between people may include, for example, the sound of hitting someone with a hand or kicking someone with a foot. Sounds of contact between people and objects may include, for example, the sound made when a person's body hits the floor when they fall or stumble, or the sound of someone sliding off a bed.

[0068] The voice recognition unit 212 extracts a human voice from the voice acquired by the acquisition unit 211, for example, using known pattern recognition technology, voiceprint analysis technology, spectrum analysis technology, etc. Furthermore, the voice recognition unit 212 converts the extracted human voice into text as a voice related to the care recipient 80, using pattern recognition technology, etc. The state determination unit 214 can determine that the voice collected by the microphone 15 is a specific voice related to the care recipient 80, if the text converted by the voice recognition unit 212 includes, for example, the specific word "help me," which is used to ask for help from others.

[0069] For example, assume that a care recipient 80 walking in a room falls and calls for help. As shown in FIG. 11 , the posture detection unit 213 detects that the care recipient 80 is in a supine position (face down) based on the captured image 602. If the care recipient 80 is in a supine position in a location other than the bed 90, there is a possibility that the care recipient 80 has fallen. However, there is a risk of erroneous determination if the care recipient 80 has fallen and is calling for help based solely on the fact that the care recipient 80 is in a supine position in a location other than the bed 90. This is because, for example, the care recipient 80 may drop something on the floor and crawl to pick it up. In such a case, the posture detection unit 213 detects that the care recipient 80 is in a supine position (face down) based on the captured image 602, but the condition of the care recipient 80 is not abnormal.

[0070] In this embodiment, the state determination unit 214 determines whether the care recipient 80 is in an abnormal state based not only on the detection result of the posture of the care recipient 80 but also on the voice related to the care recipient 80. For example, if the care recipient 80 is lying down and the voice related to the care recipient 80 that has been converted into text contains specific words asking for help from others, the state determination unit 214 determines that the care recipient 80 has fallen and is in a state of asking for help. This allows the state determination unit 214 to accurately determine whether the care recipient 80 is in an abnormal state.

[0071] Next, the notification unit 215 notifies the determination result (step S104). For example, when the condition determination unit 214 determines that the condition of the care recipient 80 is abnormal, the notification unit 215 can be configured to notify the mobile terminal 40 of the care staff 70 and / or the manager of the determination result that the condition of the care recipient 80 is abnormal. The mobile terminal 40 displays on the display that the care recipient 80 has fallen and is calling for help. This allows the care staff 70 and / or the manager, who have confirmed the message, to rush to the room of the care recipient 80 immediately.

[0072] Furthermore, when the condition determination unit 214 determines that the condition of the care recipient 80 is abnormal, the notification unit 216 can also transmit an image of the condition of the care recipient 80 determined to be abnormal to the care staff 70 and / or the mobile terminal 40 of the manager. The care staff 70 and / or the mobile terminal 40 of the manager displays the received image in real time. This allows the care staff 70 and / or the manager to check the image of the condition of the care recipient 80 determined to be abnormal in real time.

[0073] Next, the control unit 21 records the captured image in which the condition is determined to be abnormal (step S104). When the condition determination unit 214 determines that the condition of the care recipient 80 is abnormal, the control unit 21 records the captured image acquired by the acquisition unit 211 in the recording unit 24. This allows the manager or the like to hold a countermeasure meeting with the care staff 70 in charge of the care recipient 80, the manager, and other relevant parties, and while checking the captured images recorded in the recording unit 24, they can investigate the cause of the care recipient 80's abnormality and consider measures to prevent the accident from recurring. On the other hand, when the condition of the care recipient 80 is determined to be normal, the captured image is not recorded in the recording unit 24. This makes it possible to reduce the usage of the recording unit 24.

[0074] Note that the control unit 21 may be configured to store specific information that identifies the abnormality determination image in the storage unit 23 and manage the specific information, instead of recording a captured image in which the condition of the care recipient 80 is determined to be abnormal (hereinafter also referred to as an "abnormality determination image") in the recording unit 24. The storage unit 23 may store, as the specific information, for example, the date and time when the abnormality determination image was captured or the time elapsed since the start of capturing.

[0075] As described above, in the processing of the flowchart shown in FIG. 8 , the control unit 21 acquires audio from within the room where the care recipient 80 is present and a captured image including the care recipient 80, and detects the posture of the care recipient 80. Next, the control unit 21 determines whether there is an abnormality in the condition of the care recipient 80 based on the acquired audio related to the care recipient 80 and the detected posture of the care recipient 80, and notifies the control unit 21 of the determination result. For example, in FIG. 11 , a "thud" sound generated when the care recipient 80 falls, a loud cry of "help!" from the care recipient 80, and a captured image 602 including the care recipient 80 are acquired. Next, based on the captured image 602, it is detected that the posture of the care recipient 80 is lying face down. Next, based on the sounds related to the care recipient 80, such as a "thud" and a loud cry of "help," and the care recipient 80's prone position, it is determined that the person 500 has fallen and is calling for help, and the care staff 70 and / or administrator are notified.

[0076] That is, the control unit 21 determines the condition of the care recipient 80 based on the voice related to the care recipient 80 that has a level equal to or higher than a predetermined threshold and the detection result of the posture of the care recipient 80, and notifies the mobile terminal 40 of the care staff 70 or the like, and at the same time, records an image of the abnormality determination. The care staff 70 or the like can view the image (image of the abnormality determination) sent by the notification unit 215 in real time and respond on the spot, or view the image and take measures at a later date.

[0077] In the above, the processing procedure of the nursing care video analysis method of this embodiment has been explained using the example of a case where the care recipient 80 has fallen and is calling for help, but the processing procedure of the nursing care video analysis method is the same when the care recipient 80 is in another abnormal condition.

[0078] The relationship between the abnormal state of the care recipient 80, the voice related to the care recipient 80, and the posture of the care recipient 80 is summarized in Table 1 below.

[0079] [Table 1]

[0080] As shown in Table 1, the condition determination unit 214 can determine abnormal conditions of the care recipient 80, such as, but not limited to, falls, coughing, aspiration, swallowing, fright, vomiting, slipping (off the bed 90), violence, abuse, etc.

[0081] Furthermore, the "number of people" in Table 1 refers to the number of people included in the captured image. The state determination unit 214 can determine the number of people included in the captured image. The state determination unit 214 distinguishes people based on changes over time in the positions of joint points or skeletons connecting joint points detected by the posture detection unit 213, and determines the number of people included in the captured image. The changes over time in the positions of joint points or skeletons include, for example, at least one of the amount of movement and the direction of movement of the positions of joint points or skeletons. The state determination unit 214 can take into account the number of people included in the captured image when determining the state of the care recipient 80. For example, if the care recipient 80 is being subjected to violence or abuse by others, such as the care staff 70, the captured image will contain multiple people. For example, the condition determination unit 214 can determine that the care recipient 80 is being subjected to violence or abuse by others such as the care staff 70 if the captured image contains multiple people, the collected audio includes words such as "Ouch!" or sounds of people hitting or kicking each other, and the people are in a standing, sitting, or lying down position.

[0082] When it is determined that a captured image contains multiple people, the state determination unit 214 may analyze physical contact between the multiple people based on the joint points or skeletons of the multiple people. Violence and abuse are particularly likely to occur in situations where two people are alone in a closed space where it is difficult for others to see what is going on. Therefore, when it is determined that a captured image contains two people, the state determination unit 214 may be configured to determine whether specific physical contact has occurred between a first person and a second person contained in the captured image. Whether specific physical contact has occurred may be determined, for example, based on whether the body positions of the first person and the second person satisfy the following predetermined conditions:

[0083] The state determination unit 214 determines that a predetermined condition is met when the position of the hand or foot of the second person repeatedly moves to the position (area) of the first person's body (e.g., head, face, legs, etc.) a certain number of times or more in the captured image. The certain number of times may be, for example, two or more times, preferably three or more times. The positions of the first person's hand or foot and the position of the second person's body may be estimated based on the joint points or skeletons of the first person and the second person, respectively. On the other hand, the state determination unit 214 determines that the predetermined condition is not met when the position of the second person's hand or foot does not repeatedly move to the position (area) of the first person's body (e.g., head, face, legs, etc.). The state determination unit 214 also determines that the predetermined condition is not met when the number of times the position of the second person's hand or foot repeatedly moves to the position (area) of the first person's body (e.g., head, face, legs, etc.) is less than the certain number of times.

[0084] [When analyzing using pre-recorded audio and video footage] The care video analysis device may be configured to perform analysis using video (images) captured in advance by the camera 14 or an external camera and stored in the storage unit 23, rather than using images captured in real time by the camera 14. In this case, the acquisition unit 211 may be configured to acquire the captured images from the storage unit 23, instead of acquiring the captured images from the camera 14. The care video analysis device may also be configured to perform analysis using audio collected in advance by the microphone 15 or an external microphone and stored in the storage unit 23, rather than using audio collected in real time by the microphone 15.

[0085] [Identifying the person] For example, when there is one person in a room with a certain number, the control unit 21 assumes that the person in the room with the number is the care recipient 80 residing in the room with the number, and can identify the person in the room from the correspondence between the room number and the care recipient 80 residing in the room. Furthermore, even when there is a care recipient 80 and a care staff member 70 in the room, it is often possible to distinguish between the care recipient 80 and the care staff member 70 based on differences in their walking speeds and movements.

[0086] Alternatively, the control unit 21 can identify people in the room by performing facial authentication of people included in the captured image. For example, the control unit 21 estimates the joint points of the person based on the captured image and authenticates the person based on the estimation result. More specifically, the control unit 21 acquires distance values ​​between each part of the person's face based on the captured image and compares them with distance values ​​between each part of the face of a pre-registered care recipient 80, care staff 70, etc., to perform facial authentication of the person. Examples of facial parts include eyes (left and right), ears (left and right), nose, etc. Even when there are multiple people included in the captured image, the control unit 21 identifies each person as a pre-registered care recipient 80 or care staff 70 by comparing the distance values ​​between each part of the face of each person with the distance values ​​between each part of the face of the pre-registered care recipient 80 or care staff 70.

[0087] The nursing video analysis device according to the embodiment described above has the following advantages.

[0088] The condition of the care recipient 80 is determined based on the voice related to the care recipient 80 and the posture of the care recipient 80, so it is possible to accurately determine any abnormality in the condition of the care recipient 80. Furthermore, the manager or the like can hold a countermeasure meeting with the care staff 70 in charge of the care recipient 80, the manager, and other related parties, and while checking the captured images recorded in the recording unit 24, can investigate the cause of the abnormality in the condition of the care recipient 80 and consider measures to prevent the accident from recurring.

[0089] The system configuration described above is a main configuration for explaining the features of the above-described embodiment, but is not limited to the above-described configuration and can be modified in various ways within the scope of the claims.

[0090] For example, in the above example, the server 20 of the monitoring system 1 functions as a care video analysis device, but the present invention is not limited to this case. The care video analysis device may be implemented in a personal computer, server, cloud, or the like independent of the monitoring system 1, and configured to acquire video of the inside of the care recipient 80's room via a network. Also, the detection unit 10 may be configured to function as a care video analysis device.

[0091] The monitoring system 1 may also include a LIDAR (Light Detection and Ranging) in the room of the care recipient 80. The LIDAR irradiates a target object with pulses of laser light and detects the reflected light to measure the distance to the target object. The LIDAR generates a 3D map (distance image) of the body using the measured distance values ​​as pixel values ​​and transmits the distance image to the acquisition unit 211. The posture detection unit 213 may detect the skeleton of the person in the room based on the distance image.

[0092] Furthermore, the above-described flowcharts may include steps other than those shown in the flowcharts, or some steps may not be included. The order of the steps is not limited to the above-described embodiment. Furthermore, each step may be combined with other steps and executed as a single step, may be included in other steps and executed, or may be divided into multiple steps and executed.

[0093] The means and methods for performing various processes in the above-described embodiments can be realized by either dedicated hardware circuits or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage device such as a hard disk. The program may be provided as standalone application software or may be incorporated into the device's software as a function. [Explanation of symbols]

[0094] 1. Monitoring system, 10 detection unit, 11 control section, 12 Communications Department, 13 storage section, 14 cameras, 15 Mike, 16 Care Call Department, 20 servers, 21 control section, 211 Acquisition Department; 212 Speech recognition unit, 213 Attitude detection unit, 214 status determination unit, 215 Notification Department; 22 Communications Department, 23 storage section, 24 Recording section, 30 fixed terminals, 31 control section, 32 Communications Department, 33 Memory section, 34 Display section, 35 input section, 40 mobile devices, 41 control section, 42 Radio Communication Department, 43 Memory section, 44 Input display unit, 45 Audio input / output unit, 46 position detection unit, 70 care staff, 80 care recipients, 90 beds.

Claims

1. an acquisition unit that acquires audio in a room where a person is present and an image including the person; a posture detection unit that detects a posture of a person included in the image; a state determination unit that determines a state of the person based on the voice related to the person acquired by the acquisition unit and the posture of the person detected by the posture detection unit; A nursing care video analysis system having a notification unit that notifies the result of the determination by the state determination unit.

2. The nursing video analysis system of claim 1, wherein the condition determination unit determines that the person's condition is abnormal if the person's posture is maintained the same for a certain period of time and the audio acquired by the acquisition unit is a specific audio related to the person.

3. a speech recognition unit that converts speech related to the person into text; The nursing video analysis system of claim 2, wherein the state determination unit determines that the voice is a specific voice related to the person if the text converted by the voice recognition unit includes specific words that ask for help from others.

4. The nursing care video analysis system according to claim 1 or 2, wherein the same posture is any one of a sitting posture, a lying posture, a standing posture, and a posture leaning against an object.

5. The nursing video analysis system according to claim 1 , wherein the posture detection unit estimates a skeleton of the person and detects the posture of the person based on the estimated skeleton.

6. a sound collection unit installed in the room and configured to collect sounds from within the room; an imaging unit that is installed in the room and captures an image of the person in the room; The nursing care video analysis system according to claim 1 , wherein the acquisition unit acquires the sound collected by the sound collection unit and the image captured by the image capture unit.

7. The nursing video analysis system according to claim 6 , wherein the sound collection unit and the image capture unit are housed in a sensor box installed on a ceiling of the room.

8. The nursing video analysis system according to claim 1 or 2, wherein the notification unit notifies the determination result to a terminal of a care staff member providing care to the person and / or a terminal of a manager of the care staff member.

9. The nursing video analysis system according to claim 2 , wherein the specific sounds include human voices generated due to pain, surprise, or fright, or sounds of contact between people or between people and objects.

10. The nursing video analysis system according to claim 6, further comprising a recording unit that records the image acquired by the acquisition unit when the state determination unit determines that the state of the person is abnormal.

11. The nursing video analysis system according to claim 6, wherein the acquisition unit acquires the sound collected by the sound collection unit when the level of the sound collected by the sound collection unit is equal to or greater than a predetermined threshold.

12. (a) acquiring audio from a room in which a person is present and an image including the person; (b) detecting a pose of a person included in the image; a step (c) of determining a state of the person based on the voice related to the person acquired in the step (a) and the posture of the person detected in the step (b); and step (d) of notifying the result of the determination in step (c).

13. The nursing video analysis method described in claim 12, further comprising a step (e) of recording the image acquired in step (a) if the person's condition is determined to be abnormal in step (c).

14. A care video analysis program for causing a computer to execute the processes included in the care video analysis method according to claim 12 or 13.

Images