Caregiving video analysis system, caregiving video analysis method, and caregiving video analysis program

Abstract

To provide a caregiving video analysis system, a caregiving video analysis method, and a caregiving video analysis program that can detect occurrences of inappropriate behavior by a care recipient towards a caregiver, regardless of a situation in which the caregiver is placed.SOLUTION: A caregiving video analysis system includes an image acquisition unit, a skeleton detection unit, a number of people determination unit, a physical contact analysis unit, and a storage unit. The image acquisition unit acquires an image including people. The skeleton detection unit detects skeletons of the people included in the image. The number of people determination unit determines the number of people included in the image. The physical contact analysis unit, when the number of people included in the image is determined more than one by the number of people determination unit, analyzes a physical contact between a first person and a second person included in the image based on the detection result by the skeleton detection unit. The storage unit, when a specific physical contact is determined to have occurred between the first person and the second person by the physical contact analysis unit, stores an image in which the specific physical contact is determined to have occurred, or information identifying that image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a care video analysis system, a care video analysis method, and a care video analysis program. [Background technology]

[0002] Japan has seen a remarkable increase in life expectancy due to improvements in living standards, sanitary conditions, and medical standards that came with the rapid economic growth following the war. This, coupled with a declining birthrate, has led to an aging society with a high aging rate. In such an aging society, an increase in people requiring nursing care or care due to illness, injury, aging, etc. (hereinafter referred to as "care recipients") is expected. In nursing care facilities such as hospitals and elderly welfare facilities (hereinafter referred to simply as "facilities"), care is provided to care recipients by caregivers and nurses (hereinafter referred to as "care staff").

[0003] When care staff provide care to care recipients, there are cases where the care staff's body comes close to or comes into contact with the care recipient's body, such as assisting them with toileting, assisting them with transferring to a wheelchair, etc. For example, when assisting a care recipient sitting in bed to transfer to a wheelchair, the care staff often transfers the care recipient from the bed to the wheelchair by holding the care recipient in their arms.

[0004] Although care for such care recipients may be provided by multiple care staff, it is often the case that a single care staff member is in charge of providing care in the recipient's room, etc. When a single care staff member provides care for a care recipient in a space where it is difficult for others to see what is happening inside the room, such as the care recipient's room, the care staff member may be subjected to sexual harassment by the care recipient, which is a problem. Furthermore, in spaces where it is difficult for others to see what is happening inside the room, there have also been cases where the care recipient has been subjected to violence and abuse by the care staff member, which is also a problem.

[0005] In relation to this, Patent Document 1 below discloses a video recording system that is configured to continuously record images of nursing care sites, but when inappropriate behavior such as sexual harassment occurs, only a specified period of video footage is recorded before and after the occurrence. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-67508 Summary of the Invention [Problem to be solved by the invention]

[0007] However, because the video recording system described in Patent Document 1 starts recording when a change in the caregiver's heart rate or a specific sound is detected, there is a possibility that the video of the inappropriate behavior may not be recorded depending on the caregiver's situation. For example, there may be cases where the caregiver's heart rate does not change even though the care recipient's inappropriate behavior is unpleasant, or the caregiver is so surprised by the inappropriate behavior that they are unable to speak up, or the caregiver is unable to speak up because they do not want to cause trouble. In such cases, even if the caregiver later complains about the care recipient's inappropriate behavior, they are often forced to accept the situation without evidence and endure the situation.

[0008] The present invention has been made in consideration of the above circumstances, and its main purpose is to provide a care video analysis system, a care video analysis method, and a care video analysis program that can detect inappropriate behavior by a care recipient toward a caregiver, regardless of the situation the caregiver is in. [Means for solving the problem]

[0009] The above-mentioned problems of the present invention are solved by the following means.

[0010] (1) A nursing care video analysis system comprising: an image acquisition unit that acquires an image including a person; a skeleton detection unit that detects the skeletons of the people included in the image; a number determination unit that determines the number of people included in the image; a physical contact analysis unit that, when the number determination unit determines that there are multiple people included in the image, analyzes physical contact between a first person and a second person included in the image based on the detection result by the skeleton detection unit; and, when the physical contact analysis unit determines that specific physical contact occurred between the first person and the second person, a memory unit that stores an image in which it is determined that the specific physical contact occurred, or information that identifies the image.

[0011] (2) A care video analysis system as described in (1) above, further comprising an imaging unit for imaging the care recipient and the caregiver, and the image acquisition unit acquires images of the care recipient and the caregiver captured by the imaging unit.

[0012] (3) A nursing video analysis system as described in (1) or (2) above, wherein the physical contact analysis unit analyzes the physical contact between the first person and the second person when the number determination unit determines that the number of people included in the image is two.

[0013] (4) A care video analysis system as described in (1) or (2) above, further comprising a person identification unit that identifies the person included in the image, and the physical contact analysis unit analyzes the physical contact made by the first person to the second person when the person identification unit identifies the first person as a person being cared for and the second person as a caregiver.

[0014] (5) The nursing video analysis system according to (1) or (2) above, wherein the specific physical contact is related to sexual harassment.

[0015] (6) The nursing video analysis system described in (1) or (2) above, wherein the physical contact analysis unit analyzes the physical contact between the first person and the second person based on the position of the skeleton corresponding to the hand of the first person or the second person.

[0016] (7) The nursing video analysis system described in (4) above, further comprising a notification unit that notifies a mobile device of a third person when the physical contact analysis unit determines that the specific physical contact has occurred.

[0017] (8) A care video analysis system as described in (1) or (2) above, further comprising a person identification unit that identifies the person included in the image, and the physical contact analysis unit analyzes the physical contact made by the second person to the first person when the first person is identified as a person being cared for and the second person is identified as a caregiver.

[0018] (9) A nursing video analysis system as described in (1) or (2) above, in which the memory unit does not store the images until the physical contact analysis unit determines that the specific physical contact has occurred, and stores only images in which it has been determined that the specific physical contact has occurred.

[0019] (10) A nursing video analysis system as described in (2) above, further comprising a recording unit for recording images captured by the photographing unit, and the memory unit stores the date and time of the image being photographed or the time elapsed since the start of photographing as the identifying information.

[0020] (11) A method for analyzing nursing care video, comprising the steps of: (a) acquiring an image including a person; (b) detecting the skeleton of the person included in the image; (c) determining the number of people included in the image; (c) analyzing physical contact between a first person and a second person based on the detection result of (b) if it is determined in (c) that there is a plurality of people included in the image; and (d) storing an image in which it is determined that specific physical contact occurred between the first person and the second person, or information identifying the image, if it is determined in (c) that specific physical contact occurred.

[0021] (12) A nursing care video analysis program for causing a computer to execute a process including the steps of: (a) acquiring an image including a person; (b) detecting the skeleton of the person included in the image; (c) determining the number of people included in the image; (c) analyzing physical contact between a first person and a second person based on the detection result of (b) if it is determined in (c) that there is more than one person included in the image; and (d) storing an image in which it is determined that specific physical contact occurred between the first person and the second person, or information identifying the image, if it is determined in (c) that specific physical contact occurred. [Effects of the Invention]

[0022] According to the present invention, when it is determined that specific physical contact has occurred between a first person and a second person, an image in which it is determined that specific physical contact has occurred or information identifying the image is stored. Therefore, it is possible to detect inappropriate behavior by the care recipient toward the caregiver, regardless of the situation of the caregiver. [Brief explanation of the drawings]

[0023] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for purposes of illustration only and are not intended to define the limits of the invention. [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a monitoring system according to a first embodiment of the present invention. [Figure 2] 2 is a diagram showing the interior of a room in which a care recipient is staying, in which the detection unit shown in FIG. 1 is arranged. FIG. [Figure 3] 2 is a block diagram illustrating a hardware configuration of a detection unit shown in FIG. 1. FIG. [Figure 4] 2 is a block diagram illustrating an example of a hardware configuration of the server shown in FIG. 1. [Figure 5] 5 is a schematic block diagram illustrating the main functions of a control unit of the server shown in FIG. 4. [Figure 6] 2 is a block diagram illustrating a hardware configuration of the fixed terminal shown in FIG. 1. [Figure 7] FIG. 2 is a block diagram showing a hardware configuration of the mobile terminal shown in FIG. [Figure 8] 10 is a flowchart illustrating a processing procedure of a care video analysis method performed by the care video analysis device. [Figure 9A] FIG. 10 is a diagram showing an example of an image taken of assistance in transferring a care recipient from a bed to a wheelchair. [Figure 9B] This is a figure following FIG. 9A. [Figure 10A] FIG. 1 is a schematic diagram illustrating a specific physical contact (contact example 1). [Figure 10B] FIG. 10 is a schematic diagram illustrating a specific physical contact (contact example 2). [Figure 10C] FIG. 10 is a schematic diagram illustrating a specific physical contact (contact example 3). [Figure 11] FIG. 10 is a schematic diagram illustrating a specific physical contact (contact example 4). [Figure 12] FIG. 10 is a block diagram illustrating an example of the hardware configuration of a server in a monitoring system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, a nursing care video analysis system, a nursing care video analysis method, and a nursing care video analysis program according to an embodiment of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the drawings, identical elements are designated by the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0025] (First embodiment) <Monitoring System 1> FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 according to a first embodiment. The monitoring system 1 of this embodiment also functions as a care video analysis system. The monitoring system 1 includes multiple detection units 10, a server 20, a fixed terminal 30, and one or more mobile terminals 40. These are connected to each other via a network 50, such as a local area network (LAN), a telephone network, or a data communication network, either wired or wirelessly, so that they can communicate with each other. The network 50 may include a repeater that relays communication signals. In the example shown in FIG. 1, the detection units 10, the server 20, the fixed terminal 30, and the mobile terminal 40 are connected to each other via a network 50, such as a wireless LAN (for example, a LAN conforming to the IEEE 802.11 standard) that includes an access point 51, so that they can communicate with each other.

[0026] The monitoring system 1 is suitably installed in buildings such as hospitals and elderly welfare facilities (nursing care facilities). In the example shown in Fig. 1, the monitoring system 1 is installed in a building of a facility that has a plurality of rooms including a plurality of rooms and a care station in which a plurality of care recipients 80 each reside. Hereinafter, the rooms in which the care recipients 80 each reside will also be simply referred to as "rooms."

[0027] The detection units 10 are arranged in the rooms of each floor (floors (1) to (3)), which are the observation areas for the care recipients 80. In the example shown in FIG. 1, on floor (1), four detection units 10 are arranged in the rooms of care recipients 80 A, B, C, and D. Although not shown, a plurality of detection units 10 are also arranged in the rooms of care recipients 80 on other floors. Each care staff member 70 carries a mobile terminal 40. The server 20 may be an external server connected to the network 50. Furthermore, the fixed terminal 30 may be omitted, and the server 20 or the mobile terminal 40 may have its functions.

[0028] <Detection unit 10> Fig. 2 is a diagram showing the interior of a room of a care recipient 80 in which the detection unit 10 shown in Fig. 1 is arranged. Fig. 3 is a block diagram showing an example of the hardware configuration of the detection unit 10 shown in Fig. 1.

[0029] 2, the camera 14 and the body movement sensor 15 are disposed, for example, on the ceiling of the room. The camera 14 and the body movement sensor 15 may also be disposed on the upper part of a wall or attached to the bed 90. The care call unit 16 is disposed, for example, at an appropriate position on the wall.

[0030] 3, the detection unit 10 includes a control unit 11, a communication unit 12, a memory unit 13, a camera 14, a body movement sensor 15, and a care call unit 16, and these components are interconnected by a bus. Each component may be mounted in a single housing or in a separate housing.

[0031] The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The control unit 11 controls each part of the detection unit 10 and performs calculation processing according to a program.

[0032] The communication unit 12 is an interface circuit (for example, a LAN card, a wireless communication circuit, etc.) for communicating with, for example, the server 20 via a LAN.

[0033] The storage unit 13 includes a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 13 stores various programs and various data. In this embodiment, as will be described later, actions such as "getting up," "getting out of bed," and "falling" are detected as events, and images captured by the camera 14 within one minute before and after the occurrence of the event can be stored in the storage unit 13.

[0034] Camera 14 functions as a photographing unit and can photograph a photographing area including, for example, bed 90 and output the photographed image (image data). Camera 14 can be configured to include an imaging optical system and a two-dimensional imaging element. The photographed image includes a still image and a video. Camera 14 can photograph the photographing area as a video consisting of a plurality of photographed images (frames) at a frame rate of, for example, 15 fps to 30 fps. Camera 14 can be, for example, a visible light camera or a near-infrared camera. Hereinafter, the photographed image photographed by camera 14 will also be simply referred to as a "photographed image."

[0035] The body movement sensor 15 transmits microwaves to a predetermined irradiation area and receives reflected waves, thereby detecting the Doppler shift of the microwaves caused by body movement (e.g., breathing) of the care recipient 80 as body movement. The predetermined irradiation area is an area where microwaves can be irradiated onto the care recipient 80, and may be, for example, an area including part or all of the bed 90. The body movement sensor 15 detects chest movement (up and down movement of the chest) associated with the breathing of the care recipient 80, and detects abnormal slight body movement by detecting disruption of the period of the chest body movement or an amplitude of the chest body movement below a preset threshold.

[0036] The care call unit 16 includes a push-button switch and detects a care call when the care recipient 80 presses the switch. Instead of a push-button switch, a care call may be detected based on the voice of the care recipient 80 using an audio microphone. The care call unit 16 may also include a microphone and speaker for voice communication with the mobile terminal 40.

[0037] The control unit 11 recognizes the behavior of the care recipient 80 from the captured image. The behaviors to be recognized include, for example, getting up from bed, getting out of bed, getting out of a seat, falling, dropping, walking, and going out.

[0038] The control unit 11 detects image silhouettes (hereinafter referred to as "human silhouettes") from multiple captured images (e.g., video images). Human silhouettes can be detected, for example, by extracting a range of pixels where the time difference between images captured at different times is relatively large. Human silhouettes may be detected from captured images using a background subtraction method. Human silhouettes may be replaced by joint points detected from captured images using a trained neural network model. The behavior of the care recipient 80 can be detected from temporal changes in the posture (e.g., standing, sitting, and lying down) of the care recipient 80 recognized based on the human silhouette. The behavior of the care recipient 80 may also be detected from the relative positional relationship between the human silhouette and objects installed in the room, such as the bed 90. For example, "getting up" may be detected as the behavior (state) of the human silhouette crossing the area of ​​the bed 90, which is preset as a coordinate area in the captured image.

[0039] Information about the behavior of the care recipient 80 detected by the detection unit 10 can be reflected in the care record of the care recipient 80.

[0040] <Server 20> FIG. 4 is a block diagram illustrating an example of the hardware configuration of server 20. Server 20 includes a control unit 21, a communication unit 22, and a storage unit 23, which are interconnected by a bus. Server 20 functions as a video analysis device. FIG. 5 is a block diagram illustrating an example of an outline of the main functions of control unit 21 of server 20 shown in FIG. 4.

[0041] The control unit 21 includes a CPU, RAM, ROM, etc. (not shown). The control unit 21 controls each unit of the server 20 and performs calculation processing according to a nursing care video analysis program. As shown in FIG. 5, the control unit 21 functions as an image acquisition unit 211, a skeleton detection unit 212, a number of people determination unit 213, a body contact analysis unit 214, a person identification unit 215, and a notification unit 216. An overview of each function is as follows:

[0042] The image acquisition unit 211 acquires a captured image including a person. The skeleton detection unit 212 detects the skeleton of a person included in the captured image. The number of people determination unit 213 determines the number of people included in the captured image. When the number of people determination unit 213 determines that there are multiple people included in the captured image, the physical contact analysis unit 214 analyzes physical contact between a first person and a second person included in the captured image based on the detection result by the skeleton detection unit 212. When the physical contact analysis unit 214 determines that specific physical contact has occurred between the first person and the second person, it stores an image in which specific physical contact has been determined to have occurred (hereinafter simply referred to as a "contact image") in the memory unit 23. Specific physical contact is, for example, physical contact from one person to another person that is considered socially or ethically inappropriate. Such contact may include, for example, sexual harassment, violence, abuse, etc. The person identification unit 215 identifies a person included in the captured image. When the body contact analysis unit 214 determines that specific body contact has occurred, the notification unit 216 notifies the mobile device 40 of the third person. The third person may be, for example, another care staff member, a care staff leader, a manager, etc. Details of the functions of the image acquisition unit 211, the skeleton detection unit 212, the number of people determination unit 213, the body contact analysis unit 214, the person identification unit 215, and the notification unit 216 will be described later.

[0043] The communication unit 22 is configured with an interface circuit for communicating with, for example, the detection unit 10, the fixed terminal 30, the mobile terminal 40, and the like via a LAN.

[0044] The storage unit 23 is configured with an HDD, an SSD, etc., and stores an OS (Operating System), various programs such as a care video analysis program, and various data. The various data include, for example, care records for each care recipient 80, authentication information for the care staff 70 and the care recipient 80, etc. In this embodiment, the storage unit 23 also stores contact images.

[0045] <Fixed terminal 30> 6 is a block diagram illustrating an example of the hardware configuration of fixed terminal 30. Fixed terminal 30 is a PC (Personal Computer) and includes a control unit 31, a communication unit 32, a storage unit 33, a display unit 34, and an input unit 35. The basic configurations of control unit 31, communication unit 32, and storage unit 33 are similar to the configurations of the corresponding elements of server 20, so redundant explanations will be omitted.

[0046] The display unit 34 is configured with a display (for example, a liquid crystal display) and displays various types of information. The display unit 34 can display, for example, the analysis results analyzed by the control unit 21 (body contact analysis unit 214). The input unit 35 is configured with a keyboard or touch panel and accepts operational input of various types of information by an administrator or the like.

[0047] The control unit 31 can accept input from the input unit 35 of a table in which the correspondence between a unique ID that identifies the detection unit 10 and the care recipient 80 is registered, transmit the table to the server 20, and store it in the memory unit 23.

[0048] Furthermore, the control unit 31 can receive, via the input unit 35, a table in which the correspondence between the unique ID that identifies the mobile terminal 40 and the care staff 70 is registered, and transmit it to the server 20, which can then store it in the storage unit 23.

[0049] In addition, the control unit 31 can receive a table via the input unit 35 that registers the correspondence between care staff 70 and the care recipients 80 that the care staff 70 are responsible for, send it to the server 20, and store it in the memory unit 23.

[0050] Furthermore, when the care staff 70 or technical staff installs the detection unit 10 in each room (residential room), the care staff 70 or technical staff or the like associates the room number with the detection unit 10 via the fixed terminal 30. The care staff 70 or technical staff or the like also calibrates and specifies the position information of objects installed in the room, such as the bed 90, i.e., the contour information of the upward view captured by the ceiling camera 14. The care staff 70 or technical staff or the like also associates the personal IDs (name, identification number, etc.) of the care recipients 80 residing in the room with each room number. The fixed terminal 30 manages a list of care recipients, including the personal IDs, room numbers, names of staff members in charge, and units to which the care recipients 80 belong. The list of care recipients can be shared within the monitoring system 1.

[0051] <Mobile terminal 40> 7 is a block diagram showing the hardware configuration of the mobile terminal 40. The mobile terminal 40 includes a control unit 41, a wireless communication unit 42, a storage unit 43, an input / display unit 44, an audio input / output unit 45, and a position detection unit 46, which are interconnected by a bus. The mobile terminal 40 may be configured by a portable communication terminal device such as a tablet computer, a smartphone, or a mobile phone. The basic configuration of the control unit 41 is similar to the configuration of the control unit 21 of the server 20, and therefore a redundant description will be omitted. The functions of the control unit 41 will be described in detail below.

[0052] The wireless communication unit 42 communicates wirelessly with each device directly or via an access point 61 using wireless communication standards such as Wi-Fi and Bluetooth (registered trademark).

[0053] The storage unit 43 is configured with an SSD or an SD card, and stores mobile applications compatible with the monitoring system 1, various programs, and various data.

[0054] The input display unit 44 is configured by, for example, a touch panel in which a touch sensor is superimposed on a display surface of a liquid crystal display, etc. The input display unit 44 displays various information and receives various inputs.

[0055] The audio input / output unit 45 is, for example, a speaker and a microphone, and enables the care staff 70 to make audio calls to other mobile terminals 40 via the wireless communication unit 42. The audio input / output unit 45 enables the care staff 70 to make audio calls to the care recipient 80 via the wireless communication unit 42 and the detection unit 10 via the server 20.

[0056] The position detection unit 46 is a GPS module or the like, and detects the position of the mobile terminal 40. As will be described later, the position of the mobile terminal 40 may be detected by the control unit 41 using signals transmitted from beacons installed at predetermined installation positions.

[0057] <Functions of the control unit 21> Fig. 8 is a flowchart for explaining the processing steps of a care video analysis method using a care video analysis device. The processing of the flowchart shown in the figure is realized by the CPU of the control unit 21 executing a care video analysis program. Fig. 9A is a diagram showing an example of an image captured of assistance in transferring a care recipient 80 from a bed 90 to a wheelchair. Fig. 9B is a diagram subsequent to Fig. 9A. More specifically, Fig. 9A is an image captured before transferring the care recipient 80 to the wheelchair, and Fig. 9B is an image captured a predetermined time after the image of Fig. 9A, while the care recipient is being transferred to the wheelchair.

[0058] [When recording the interior of 80 care recipients' rooms in real time and analyzing the footage] Assume that care staff 70 is providing care to care recipient 80 in the care recipient's room. The care provided by care staff 70 to care recipient 80 is captured by camera 14 of detection unit 10. As described above, camera 14 captures an image of a capture area including, for example, bed 90, and outputs the captured image. The capture area includes at least care staff 70 and care recipient 80.

[0059] As shown in Fig. 8, the image acquisition unit 211 acquires an image including a person (step S101). As shown in Figs. 9A and 9B, the captured image acquired by the image acquisition unit 211 includes, for example, a first person 81 and a second person 71. Fig. 9A shows the first person 81 trying to get off the bed 90 while holding on to it. The second person 71 is watching the movements of the first person 81. Fig. 9B also shows the first person 81 being supported by the second person 71 while moving backward toward a wheelchair.

[0060] Next, the skeleton detection unit 212 detects the skeletons of the people included in the captured image (step S102). The skeleton detection unit 212 estimates the joint points of the first person 81 and the second person 71 included in the captured image. In FIGS. 9A and 9B, the estimated joint points JP are indicated by "●". The joint points estimated by the skeleton detection unit 212 include, for example, eyes (left and right), ears (left and right), nose, elbows (left and right), wrists (left and right), shoulders (left and right), neck, waist (left and right), pelvis, knees (left and right), and ankles (left and right).

[0061] For example, the skeleton detection unit 212 can detect a human rectangle using a trained model of a neural network that has been trained to estimate a human rectangle containing a person from a captured image. The skeleton detection unit 212 can then detect the joint points of the person using a trained model of a neural network that has been trained to estimate joint points from a human rectangle. An example of a trained model for estimating a human rectangle from a captured image is the RPN (Region Proposal Network) model. Also, an example of a trained model for detecting the joint points of a person from a human rectangle is a model such as Deep Pose, CNN (Convolution Neural Network), or Res Net.

[0062] Next, the number of people determination unit 213 determines the number of people included in the image (step S103). The number of people determination unit 213 distinguishes people and determines the number of people included in the captured image based on changes over time in the positions of the joint points or skeletons connecting the joint points detected by the skeleton detection unit 212. The changes over time in the positions of the joint points or skeletons include, for example, at least one of the amount of movement and the direction of movement of the positions of the joint points or skeletons.

[0063] Next, the body contact analysis unit 214 analyzes the physical contact between people (step S104). When the number of people determination unit 213 determines that there are multiple people included in the captured image, the body contact analysis unit 214 can analyze the physical contact between the multiple people based on the joint points or skeletons of the multiple people. In particular, the above-mentioned inappropriate behavior is likely to occur in situations where there are only two people in an enclosed space where it is difficult for others to see what is going on. Therefore, when it is determined that there are two people included in the captured image, the body contact analysis unit 214 can be configured to determine whether specific physical contact has occurred between the first person 81 and the second person 71. Whether specific physical contact has occurred can be determined based on, for example, whether the body position of the first person 81 and the body position of the second person 71 satisfy a predetermined condition. The predetermined condition will be described in detail later.

[0064] Next, when it is determined that specific physical contact has occurred between the first person 81 and the second person 71, the body contact analysis unit 214 stores the contact image in the memory unit 23. That is, until it is determined that specific physical contact has occurred, the body contact analysis unit 214 does not store the captured image in the memory unit 23, but stores only the contact image in the memory unit 23. The memory unit 23 stores the contact image (step S105).

[0065] Next, the notification unit 216 notifies the result of the analysis (step S106). When the physical contact analysis unit 214 determines that specific physical contact has occurred, the notification unit 216 notifies the mobile device 40 of the third person. For example, the notification unit 216 may display a message on the display of the mobile device 40 of the third person, indicating that there is a possibility of sexual harassment from the first person 81 to the second person 71. This allows the third person, who has confirmed the message, to rush to the room of the care recipient 80.

[0066] Furthermore, when the physical contact analysis unit 214 determines that specific physical contact has occurred, the notification unit 216 can also transmit an image in which it has been determined that specific physical contact has occurred to the mobile device 40 of the third person. The mobile device 40 of the third person displays the received image in real time. This allows the third person to check the image in which it has been determined that specific physical contact has occurred in real time.

[0067] 8, a captured image including a person is acquired, and the skeletons of the person included in the captured image are detected. The number of people included in the captured image is determined, and if it is determined that the captured image includes multiple people, physical contact between the first person 81 and the second person 71 is analyzed based on the detection results of the person skeletons. If it is determined that specific physical contact has occurred between the first person 81 and the second person 71, a contact image is stored in the storage unit 23.

[0068] [When analyzing pre-recorded footage] The nursing care video analysis device may be configured to perform analysis using video (images) that have been captured in advance by the camera 14 or an external camera and stored in the storage unit 23, rather than using images captured in real time by the camera 14. In this case, the image acquisition unit 211 may be configured to acquire the captured images from the storage unit 23 instead of acquiring the captured images from the camera 14.

[0069] [Specified conditions] When any of the following first to third conditions is met, the physical contact analysis unit 214 determines that specific physical contact has occurred and that the specific physical contact is related to sexual harassment.

[0070] First condition As shown in FIG. 10A , the body contact analysis unit 214 determines that the first condition is met when the hand position of the first person 81 is in the position (area) of the lower body (lower abdomen or buttocks) of the second person 71 for a certain period of time or longer. The certain period of time may be, for example, 30 seconds or longer, preferably 60 seconds or longer. The hand positions of the first person 81 and the lower body position of the second person 71 may be estimated based on the joint points or skeletons of the first person 81 and the second person 71, respectively. On the other hand, the body contact analysis unit 214 determines that the first condition is not met when the hand position of the first person 81 is not in the position (area) of the lower body of the second person 71, or when the time that the hand position of the first person 81 is in the position (area) of the lower body of the second person 71 is shorter than the certain period of time.

[0071] Second condition As shown in FIG. 10B , the body contact analysis unit 214 determines that the second condition is met if the position of the hand of the first person 81 is in the position (area) of the chest of the second person 71 for a certain period of time or longer. The certain period of time may be, for example, 30 seconds or longer, preferably 60 seconds or longer. The positions of the hand of the first person 81 and the position of the chest of the second person 71 may be estimated based on the joint points or skeletons of the first person 81 and the second person 71, respectively. On the other hand, the body contact analysis unit 214 determines that the second condition is not met if the position of the hand of the first person 81 is not in the position (area) of the chest of the second person 71, or if the time that the position of the hand of the first person 81 is in the position (area) of the chest of the second person 71 is shorter than the certain period of time.

[0072] Third condition As shown in FIG. 10C , the body contact analysis unit 214 determines that the third condition is met if the head position of the first person 81 is in the head position (area) of the second person 71 for a certain period of time or longer. The certain period of time may be, for example, 30 seconds or longer, preferably 60 seconds or longer. The head positions of the first person 81 and the second person 71 may be estimated based on the joint points or skeletons of the first person 81 and the second person 71, respectively. On the other hand, the body contact analysis unit 214 determines that the third condition is not met if the head position of the first person 81 is not in the head position (area) of the second person 71, or if the time that the head position of the first person 81 is in the head position (area) of the second person 71 is shorter than the certain period of time.

[0073] Furthermore, if the following fourth condition is met, the physical contact analysis unit 214 determines that specific physical contact has occurred and that the specific physical contact is related to violent or abusive behavior.

[0074] Fourth condition As shown in FIG. 11 , the body contact analysis unit 214 determines that the fourth condition is met when the position of the hands or feet of the second person 71 repeatedly moves to the position (area) of the body (e.g., head, face, legs, etc.) of the first person 81 a certain number of times or more. The certain number of times may be, for example, two or more times, preferably three or more times. The positions of the hands or feet of the first person 81 and the position of the body of the second person 71 may be estimated based on the joint points or skeletons of the first person 81 and the second person 71, respectively. On the other hand, the body contact analysis unit 214 determines that the fourth condition is not met when the position of the hands or feet of the second person 71 does not repeatedly move to the position (area) of the body (e.g., head, face, legs, etc.) of the first person 81. In addition, the body contact analysis unit 214 also determines that the fourth condition is not met if the number of times the position of the hand or foot of the second person 71 repeatedly moves to the position (area) of the body (e.g., head, face, legs, etc.) of the first person 81 is less than a certain number of times.

[0075] [Identifying the person] The person identification unit 215 performs facial authentication of the first person 81 and the second person 71, and can identify the first person 81 and the second person 71. The person identification unit 215 authenticates the first person 81 and the second person 71 whose joint points have been estimated based on the captured image. For example, the person identification unit 215 identifies that the multiple first persons 81 and second persons 71 included in the image are care recipients 80 and care staff 70, respectively, that have been registered in advance. More specifically, the person identification unit 215 acquires distance values ​​to each part of the person's face (for example, eyes (left and right), ears (left and right), nose, etc.) based on the captured image, and performs facial authentication of the person.

[0076] The nursing video analysis device according to the first embodiment described above has the following advantages.

[0077] When it is determined that specific physical contact has occurred between the first person 81 and the second person 71, a contact image is stored. Therefore, it is possible to detect inappropriate behavior by the care recipient 80 toward the care staff 70, regardless of the situation in which the care staff 70 is placed. This allows the manager, etc. to warn the care recipient 80 to stop the inappropriate behavior by showing the contact image to the care recipient 80 as evidence, if necessary. In addition, the manager, etc. can hold a countermeasure meeting with the relevant parties and, while reviewing the contact image, consider measures to prevent recurrence.

[0078] (Second embodiment) In the first embodiment, a case has been described in which a contact image is stored when it is determined that specific physical contact has occurred between a first person 81 and a second person 71. In the second embodiment, when it is determined that specific physical contact has occurred between a first person 81 and a second person 71, information (specific information) that identifies an image (contact image) in which it has been determined that specific physical contact has occurred is stored. In the following, to avoid duplication of explanation, detailed explanations of the same configuration as in the first embodiment will be omitted.

[0079] FIG. 12 is a block diagram illustrating an example of the hardware configuration of a server in a monitoring system according to the second embodiment. In this embodiment, the care video analysis device further includes a recording unit 24 that records images captured by the camera 14. The recording unit 24 has a sufficient capacity to record images captured by the camera 14 for a predetermined period of time. The recording unit 24 may be provided external to the care video analysis device. The predetermined period may be approximately one day to several days, depending on the frequency at which the manager checks the analysis results obtained by the body contact analysis unit 214.

[0080] When it is determined that specific physical contact has occurred between the first person 81 and the second person 71, the body contact analysis unit 214 stores information identifying the contact image in the storage unit 23. The storage unit 23 may store, as the identification information, for example, the date and time when the contact image was captured or the time elapsed since the start of capturing the image.

[0081] The nursing video analysis device according to the second embodiment described above has the following advantages.

[0082] The photographing date and time of the contact image or the elapsed time from the start of photographing is stored in the storage unit 23. Therefore, for photographed images recorded in the recording unit 24, by specifying the photographing date and time of the contact image or the elapsed time from the start of photographing, the contact image can be read out from the recording unit 24.

[0083] The system configuration described above is a main configuration for explaining the features of the above-described embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims.

[0084] For example, in the above example, the server 20 of the monitoring system 1 functions as a care video analysis device, but the present invention is not limited to this case. The care video analysis device may be implemented in a personal computer, server, cloud, or the like independent of the monitoring system 1, and configured to acquire video of the inside of the care recipient 80's room via a network. Also, the detection unit 10 may be configured to function as a care video analysis device.

[0085] The monitoring system 1 may also include a LIDAR (Light Detection and Ranging) in the room of the care recipient 80. The LIDAR irradiates a target object with pulses of laser light and detects the reflected light to measure the distance to the target object. The LIDAR generates a 3D map (distance image) of the body using the measured distance values ​​as pixel values ​​and transmits the distance image to the image acquisition unit 211. The skeleton detection unit 212 may detect the skeleton of a person in the room based on the distance image.

[0086] Furthermore, the above-described flowcharts may include steps other than those shown in the flowcharts, or some steps may not be included. The order of the steps is not limited to the above-described embodiment. Furthermore, each step may be combined with other steps and executed as a single step, may be included in other steps and executed, or may be divided into multiple steps and executed.

[0087] The means and methods for performing various processes in the above-described embodiments can be realized by either dedicated hardware circuits or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage device such as a hard disk. The program may be provided as standalone application software or may be incorporated into the device's software as a function.

[0088] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims. [Explanation of symbols]

[0089] 1. Monitoring system, 10 detection unit, 11 control section, 12 Communications Department, 13 storage section, 14 cameras, 15 Body movement sensors, 16 Care Call Department, 20 servers, 21 control section, 211 image acquisition unit, 212 skeleton detection unit, 213 Number of people determination department, 214 Body contact parts, 215 Person Identification Department; 216 Notification Department; 22 Communications Department, 23 storage section, 24 Recording section, 30 fixed terminals, 31 control section, 32 Communications Department, 33 Memory section, 34 Display section, 35 input section, 40 mobile devices, 41 control section, 42 Radio Communication Department, 43 Memory section, 44 Input display unit, 45 Audio input / output unit, 46 position detection unit, 70 care staff, 71 Second Person, 80 care recipients, 81 First Person, 90 beds.

Claims

1. an image acquisition unit that acquires an image including a person; a skeleton detection unit that detects the skeleton of a person included in the image; a number-of-people determination unit that determines the number of people included in the image; a body contact analysis unit that analyzes body contact between a first person and a second person included in the image based on a detection result by the skeleton detection unit when the number-of-persons determination unit determines that there are multiple people included in the image; and A nursing care video analysis system comprising: a memory unit that, when the physical contact analysis unit determines that specific physical contact occurred between the first person and the second person, stores an image in which the specific physical contact is determined to have occurred, or information identifying the image.

2. The device further includes an imaging unit for imaging the care recipient and the caregiver, The care video analysis system according to claim 1 , wherein the image acquisition unit acquires images of the care recipient and the caregiver captured by the imaging unit.

3. The nursing video analysis system of claim 1 or 2, wherein the physical contact analysis unit analyzes physical contact between the first person and the second person when the number determination unit determines that the number of people included in the image is two.

4. a person identification unit for identifying a person included in the image; The nursing video analysis system of claim 1 or 2, wherein the physical contact analysis unit analyzes physical contact made by the first person to the second person when the person identification unit identifies the first person as a person being cared for and the second person as a caregiver.

5. The nursing care video analysis system according to claim 1 or 2, wherein the specific physical contact is related to sexual harassment.

6. The nursing video analysis system of claim 1 or 2, wherein the physical contact analysis unit analyzes physical contact between the first person and the second person based on the position of a skeleton corresponding to the hand of the first person or the second person.

7. The nursing video analysis system of claim 4, further comprising a notification unit that notifies a mobile device of a third person when the physical contact analysis unit determines that the specific physical contact has occurred.

8. a person identification unit for identifying a person included in the image; The nursing video analysis system of claim 1 or 2, wherein the physical contact analysis unit analyzes physical contact made by the second person to the first person when the person identification unit identifies the first person as a person being cared for and the second person as a caregiver.

9. The nursing video analysis system of claim 1 or 2, wherein the memory unit does not store the images until the physical contact analysis unit determines that the specific physical contact has occurred, and stores only images that are determined to have the specific physical contact.

10. a recording unit that records the image captured by the imaging unit; The nursing video analysis system according to claim 2 , wherein the storage unit stores, as the specifying information, a date and time of the image being captured or a time elapsed since the start of the image being captured.

11. (a) acquiring an image including a person; (b) detecting a skeleton of a person included in the image; (c) determining the number of people in the image; a step (c) of analyzing physical contact between a first person and a second person included in the image based on the detection result of the step (b) when it is determined in the step (c) that the image includes a plurality of people; A nursing care video analysis method including a step (d) of storing an image in which it is determined that specific physical contact occurred between the first person and the second person, or information identifying the image, if it is determined in step (c) that specific physical contact occurred between the first person and the second person.

12. (a) acquiring an image including a person; (b) detecting a skeleton of a person included in the image; (c) determining the number of people included in the image; a step (c) of analyzing physical contact between a first person and a second person included in the image based on the detection result of the step (b) when it is determined in the step (c) that the image includes a plurality of people; A nursing care video analysis program for causing a computer to execute processing including: if it is determined in step (c) that specific physical contact has occurred between the first person and the second person, step (d) of storing an image in which it is determined that specific physical contact has occurred, or information identifying the image.

Images