Nursing system, method for watching, and control program

Abstract

To properly grasp the situations of residents, while protecting privacy of the residents to take care of.SOLUTION: A nursing system 1000 includes: a first acquisition unit 111 for acquiring bone structure information on the basis of an image of a resident; a second acquisition unit 112 for acquiring resident information showing the state of the resident; an image generation unit 114 for generating an avatar image of the resident on the basis of the resident information and the bone structure information; and an output unit 115 for displaying the avatar image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a care system, a monitoring method, and a control program. [Background technology]

[0002] In nursing care facilities such as hospitals and elderly care facilities, there is a risk that the people being watched over, such as those requiring care, may fall while walking, fall out of bed and injure themselves, or wander off and go missing. Therefore, development is underway to enable nurses or caregivers to rush to the scene immediately when the people being watched over fall into such a state, or to monitor the condition of the people being watched over to prevent them from going missing.

[0003] These monitors may display images captured by cameras so that facility staff can check the situation. In such cases, protecting the privacy of the person being monitored becomes an issue, as more information than necessary is included.

[0004] To address this issue, the technology disclosed in Patent Document 1 creates a 3D animation using a humanoid avatar in accordance with the movements of the person being watched over. Specifically, the watching system in Patent Document 1 generates skeletal movement data by detecting the skeletal movements of the person being watched over using a multi-viewpoint motion capture device. Then, a 3D body image is superimposed on this skeletal movement data to create a 3D animation image using a humanoid avatar, which is displayed to the watcher in a remote location. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-210547 Summary of the Invention [Problem to be solved by the invention]

[0006] In many cases, facilities have multiple residents, i.e., people to be monitored. However, with the technology of Patent Document 1, all monitored individuals are displayed as similar humanoid avatars. As a result, the humanoid avatars are not appropriate for each resident. For example, even if a resident has a minor leg injury, a humanoid avatar similar to the other residents is generated and displayed. When caregivers, such as caregivers, assess a resident's condition from the displayed image of the humanoid avatar, they may interpret the resident's movements as if they are uninjured, just like the other residents. In such a case, the normal response would be to determine that a resident with a leg injury needs assistance when they see the resident attempting to stand up on the monitor and rush to the resident's room to provide assistance. However, when staff see the displayed image of a humanoid avatar generated using technology such as Patent Document 1, they may mistakenly determine that assistance is not required, even when assistance is actually required. Conversely, treating all residents, including those who do not require assistance, in the same way could lead to excessive care.

[0007] The present invention has been made in consideration of the above circumstances, and aims to properly grasp the situation of a resident while protecting the privacy of the resident being watched over. [Means for solving the problem]

[0008] The above object of the present invention can be achieved by the following means.

[0009] (1) a first acquisition unit that acquires human skeleton information based on an image of a resident; a second acquisition unit that acquires resident information indicating the status of the resident; an image generation unit that generates an avatar image of the resident based on the resident information and the skeleton information; an output unit that displays the avatar image; A care system equipped with:

[0010] (2) The resident information includes at least one of ADL information, level of care required, dementia level, and presence or absence of disease, The nursing care system according to (1) above, wherein the image generation unit generates the avatar image that reflects the resident information.

[0011] (3) The nursing care system according to (1) or (2), wherein the image generation unit generates an image in which the avatar image is superimposed on a real-life image taken of the inside of a room.

[0012] (4) an imaging unit that photographs the resident; a skeletal information generating unit that generates skeletal information from the two-dimensional image obtained from the imaging unit; The care system according to (1) above, comprising:

[0013] (5) The skeletal information includes joint information relating to positions of a plurality of joint points of the person and connections of the joint points; The nursing care system according to (1) above, wherein the image generation unit generates the avatar image from the joint information.

[0014] (6) The nursing care system according to (5) above, wherein the image generating unit indexes walking speed, unsteadiness during walking, and joint bending angle from the joint information and reflects the indexes in the avatar image.

[0015] (7) The resident information includes a nursing care record that documents care regarding the resident; The care system according to (1) above, wherein the image generation unit reflects information of the care record in the avatar image.

[0016] (8) If it is detected that there are multiple people in the image, The nursing care system according to (1) above, wherein the image generation unit generates an avatar for each of a plurality of people and generates an image with an annotation indicating the person as a resident or non-resident.

[0017] (9) A storage unit for storing the image, When displaying the image stored in the storage unit, The nursing care system according to (3) above, wherein the image generation unit generates an avatar image of the resident.

[0018] (10) A reception unit that receives a user's setting of whether to display or hide the avatar; The nursing care system described in (9) above, wherein when the reception unit receives a setting to hide the image, when displaying the image stored in the memory unit, the image generation unit displays the image without generating an avatar image of the resident.

[0019] (11) A reception unit that receives display settings from a user, The nursing care system according to (1) above, wherein the image generating unit generates a synthesized image using resident information of the resident as additional information in accordance with a display setting.

[0020] (12) Step (a) of acquiring human skeleton information based on an image of the resident; (b) acquiring resident information indicating the status of the resident; (c) generating an avatar image of the resident based on the resident information and the skeletal information; (d) displaying the avatar image; Monitoring methods, including:

[0021] (13) The resident information includes at least one of ADL information, level of care required, dementia level, and presence or absence of disease, The monitoring method according to (12) above, wherein in the step (c), the avatar image is generated to reflect the resident information.

[0022] (14) Step (a) of acquiring human skeleton information based on an image of the resident; (b) acquiring resident information indicating the status of the resident; (c) generating an avatar image of the resident based on the resident information and the skeletal information; (d) displaying the avatar image; A control program for causing a computer to execute a process including the above.

[0023] (15) The resident information includes at least one of ADL information, level of care required, dementia level, and presence or absence of disease, The control program according to (14) above, wherein in the step (c), the avatar image is generated to reflect the resident information. [Effects of the Invention]

[0024] The nursing care system of the present invention includes a first acquisition unit that acquires skeletal information of a person based on an image of the resident, a second acquisition unit that acquires resident information indicating the condition of the resident, an image generation unit that generates an avatar image of the resident based on the resident information and the skeletal information, and an output unit that displays the avatar image. This makes it possible to appropriately grasp the condition of the resident while protecting the privacy of the resident being watched over. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram showing a care system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a detection unit. [Figure 3] FIG. 2 is a block diagram showing the configuration of a terminal device. [Figure 4] FIG. 1 is a block diagram showing a configuration of an information processing device. [Figure 5] FIG. 2 is a schematic diagram illustrating an avatar model. [Figure 6] FIG. 10 is a schematic diagram showing an example of an avatar model based on resident information. [Figure 7] 10 is a flowchart illustrating a monitoring method using an information processing device. [Figure 8]8 is a subroutine flowchart showing the processing of step S03 in FIG. 7. [Figure 9] FIG. 10 is a schematic diagram showing the process of step S31. [Figure 10] 10 is an example of a display image. [Figure 11] 10 is an example of a display image. [Figure 12] 10 is an example of a display image. [Figure 13] 8 is a subroutine flowchart showing the processing of step S03 in FIG. 7 in the playback mode. [Figure 14] 10 shows examples of various display images according to settings. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0027] FIG. 1 is a schematic diagram illustrating a nursing care system 1000 according to a first embodiment. The nursing care system 1000 includes an information processing device 10, a detection unit 20, and a terminal device 30, which are connected to each other via a network 50 for communication. The information processing device 10 is a server or a personal computer (PC). When the information processing device 10 is configured as a server, it may be an on-premise server installed in a nursing care facility or a cloud server using a commercial cloud service. The terminal device 30 is a PC and is operated by users such as nursing care staff (hereinafter simply referred to as staff) or administrators at the nursing care facility. The terminal device 30 may also be a mobile device such as a smartphone used by the staff. An avatar image based on skeletal information of the person being monitored (resident 71) is displayed on the terminal device 30, and the staff or administrator can understand the situation or state of the resident 71 from the avatar image. Below, the detection unit 20 will be described first, followed by the terminal device 30 and the information processing device 10. 1, a detection unit 20 and a terminal device 30 are arranged in each of a plurality of rooms in a nursing care facility. An example is shown in which each device in the nursing care facility is connected to an information processing device 10 via a network 50. That is, each device 20, 30 in one nursing care facility is connected to the information processing device 10. However, this is not limited to this, and one information processing device 10 may be connected to a plurality of nursing care facilities, each of which has a terminal device 30 and a plurality of detection units 20.

[0028] (Detection unit 20) FIG. 2 is a block diagram showing the configuration of the detection unit 20. Referring to FIGS. 1 and 2, the detection unit 20 is configured to monitor the movements (behavior) of the resident 71 in the observation area (room) of a room where the resident 71 residing in the nursing care facility is located, with the observation area being the interior of the room. The resident 71 is a person receiving care and residing in the nursing care facility. Care or nursing care is provided to the resident 71 by staff. The room is, for example, one room in a nursing care facility where multiple residents 71 reside. Furthermore, the information processing device 10 is connected to the detection unit 20 provided in each of the multiple rooms as shown in FIG. 1. The device ID of each detection unit 20 and the subject ID of the resident 71 to be monitored are associated and stored in a memory unit (memory unit 32 or memory unit 12 described below).

[0029] As shown in FIG. 2, the detection unit 20 includes a control unit 21, a communication unit 22, a camera 23, a sensor 24, a care call unit 25, etc. The detection unit 20 generates sensing data (image data, audio data, etc.) by constantly sensing the occupant 71 in the observation area (room) in real time, 24 hours a day, using a plurality of various sensors. The control unit 21 may also include a large-capacity memory. The camera 23 and the sensor 24 are disposed in the main body of the detection unit 20. The main body is disposed on the ceiling or upper part of the wall of the room. The care call unit 25 is disposed separately from the main body and is communicatively connected to the main body via wired or short-range wireless communication.

[0030] The control unit 21 is composed of a CPU, RAM, ROM, etc., and controls each part of the detection unit 20 and performs calculation processing according to a program. The communication unit 22 is an interface circuit (for example, a LAN card, etc.) for communicating with other devices such as the terminal device 30 via the network 50.

[0031] The camera 23 corresponds to an imaging unit, captures images of the room in which the resident is present, and outputs a two-dimensional image (captured image). For example, the camera 23 is placed on the ceiling or the upper part of the wall of the room and captures an overhead image of the room with the resident 71 directly below as an observation area (captured area). The camera 23 is an example of a sensor that detects the movement of the resident 71 within the observation area. Objects such as a bed 81 and a wheelchair 82 are present in the room. The camera 23 captures images of the resident 71, who is a person, as well as surrounding objects, and outputs captured data (video). The camera 23 is a near-infrared camera, but a visible light camera may be used instead, or a combination of these may be used. The camera 23 may also be a wide-angle camera. The focal length, pixel size, optical axis direction, and distortion of the optical system of the camera 23 are registered in advance and stored in the memory unit 12. Furthermore, calibration is performed to associate the real space in the room with the camera coordinate system. For example, calibration is performed by the user inputting the distance and direction to multiple markers captured by the fixed camera 23.

[0032] The sensor 24 detects the movement of the resident 71. The sensor 24 includes various sensors other than the camera 23. For example, the sensor 24 may include at least one of a body movement sensor, a bed sensor, a mat sensor, a thermal sensor, and an infrared sensor. The "body movement sensor" may be a Doppler shift sensor that transmits and receives microwaves to and from the bed 81 and detects the Doppler shift of the microwaves caused by the body movement (e.g., breathing) of the resident 71. This body movement sensor detects chest movement (up and down movement of the chest) associated with the breathing of the resident 71. The period and amplitude of the body movement can be used to determine the sleep state or abnormal micro-movements (e.g., due to cardiac arrest). The "bed sensor" may be a sensor attached to the bed. For example, the bed sensor may detect weight and be placed on the bed 81 or on the floor at the exit of the bed 81 as an observation area to detect whether a person is standing on the sensor or to detect the sleep state. The "mat sensor" has the same function as the bed sensor, detecting the presence of a person in each divided area of the floor. The "infrared sensor," also known as a human presence sensor, detects whether a person is present in the room. For example, the infrared sensor is placed throughout the room or on the bed 81 as its observation area.

[0033] The care call unit 25 includes a push button switch, and detects a care call when the switch is operated by the resident 71.

[0034] (Terminal device 30) FIG. 3 is a block diagram showing the configuration of the terminal device 30. As shown in FIG. 3, the terminal device 30 includes a control unit 31, a memory unit 32, a communication unit 33, a display unit 34, and an operation input unit 35. The control unit 31 is composed of a CPU, RAM, ROM, etc., and controls each unit of the terminal device 30 and performs calculations according to a program. The memory unit 32 is composed of a hard disk or the like that stores various programs and data. The communication unit 33 is an interface circuit for communicating with other devices via a network 50, either wired or wirelessly. The display unit 34 is composed of an LCD display, a touch panel, etc., and displays various information and operation screens. The operation input unit 35 is an input device such as a keyboard, mouse, or touch sensor, and receives input from staff, staff leaders of units (also referred to as areas or groups in charge), and facility managers (hereinafter collectively referred to as managers, etc.).

[0035] (Information processing device 10) The configuration of the information processing device 10 will be described below with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device.

[0036] 4, the information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is composed of a CPU, RAM, ROM, etc., and controls each unit of the information processing device 10 and performs calculation processing according to a program.

[0037] (Control unit 11) The control unit 11 functions as a first acquisition unit 111 and a second acquisition unit 112 in cooperation with the communication unit 13. The control unit 11 also functions as a skeletal information generation unit 113, an image generation unit 114, an output unit 115, and a setting reception unit 116. Each function of the control unit 11 will be described later. The communication unit 13 is an interface circuit for communicating with other devices via the network 50.

[0038] (Storage unit 12) The storage unit 12 is composed of a hard disk or the like that stores various programs and various data. The storage unit 12 stores basic resident data, staff information, care records, LIFE data, image data, voice data, trained models, etc. The basic resident data, care records, and LIFE data related to the resident 71 are collectively referred to as resident information.

[0039] The "resident basic data" is information about the resident 71, including the room number occupied by the resident 71, family information, the resident 71's age, gender, medical history, and medical conditions. The resident basic data also includes physical condition information and information about the resident's lifestyle. Examples of the physical condition information include the resident 71's height, weight, and level of care required. Physical conditions may also include the resident's level of independent walking and the level of ability to transfer from bed to a care chair. The physical condition information may also include the resident's dementia level. Dementia levels include mild, moderate, and severe. The dementia level is diagnosed by a doctor or may be input by a facility manager or the like. Alternatively, the terminal device 30 may determine the resident 71's level of unsteadiness when moving from images captured by the camera 23, and use this information to evaluate the dementia level. Information about the resident's lifestyle may include the resident's activity level (travel time, travel distance), sleep time (daily rhythm), number of care calls, etc. Medical history is past information about medical events and health, such as past illnesses, injuries, surgeries, treatments, and allergies.

[0040] "Staff information" includes information such as staff name, affiliated unit, and work schedule.

[0041] "Nursing care records" include nursing record sheets, nursing record sheets (medical charts, life records), and diagnostic charts that record the care and treatment provided by staff, caregivers, nurses, and doctors (hereinafter referred to as "staff, etc.") who provide care to the resident 71. These records may be entered as text by scanning sheets handwritten by the staff, etc. and performing OCR processing. Alternatively, the records may be entered by the staff, etc., using a staff terminal such as a smartphone that the staff, etc., carries while on duty. The care records record the amount of medication given to the resident 71, medication care related to medication amounts, the amount of food eaten by the resident 71, dietary and hydration care related to fluid intake, the amount and quality of excretion, vital values (body temperature, blood pressure, pulse rate, etc.), and the time of occurrence of these events.

[0042] LIFE is a scientific nursing care information system promoted by the Ministry of Health, Labor and Welfare. "LIFE data" refers to input data related to LIFE. LIFE data includes many input items such as medical history, ADL (Activities of Daily Living), and nutrition.

[0043] The image data is captured and acquired by the camera 23 of the detection unit 20 in each room. The image data is stored for a predetermined period of time. The stored image data may be played back by the terminal device 30.

[0044] The trained model is used by the skeleton information generation unit 113 described below to recognize objects such as people in captured images and to generate skeleton information for the recognized people.

[0045] (Control unit 11) (First acquisition unit 111, second acquisition unit 112) The first acquisition unit 111 acquires, from the detection unit 20, a captured image obtained by a camera 23 arranged to capture an image of the inside of a room where a resident is present. The first acquisition unit 111 also acquires skeletal information of a person based on the captured image. The skeletal information is information generated by the skeletal information generation unit 113 or information generated by a skeletal information generation unit of another device such as the detection unit 20.

[0046] The second acquisition unit 112 acquires the resident information from the terminal device 30 or the storage unit 12 .

[0047] The resident information acquired by the second acquisition unit 112 includes information indicating the resident's condition related to the avatar model. For example, the resident's condition includes information regarding physical characteristics and / or functional disabilities. For example, the resident information includes the above-mentioned resident basic data, care records, and LIFE data. In addition, the resident information includes at least one of ADL information, level of care required, dementia level, and the presence or absence of a disease regarding the resident 71.

[0048] ADL information includes assessment results for activities of daily living (getting up and down, transferring, moving, eating, dressing, toileting, bathing, and grooming). The level of care required is assessed based on the long-term care insurance system, and is rated on a scale of 1 or 2 for assistance required, or 1 to 5 for care required. Dementia level is assessed as mild, moderate, or severe. Dementia level may be determined from the level of care required. ADL information, level of care required, and dementia level are included in the resident basic data and / or LIFE data.

[0049] The presence or absence of illness includes the presence and level of traumatic and non-traumatic (non-traumatic) impairments. For example, bleeding in the legs (traumatic) or pain in the legs that interferes with walking (non-traumatic). The presence or absence of illness is included in the resident basic data, LIFE data, or nursing care records.

[0050] (Skeletal information generation unit 113) The skeletal information generation unit 113 analyzes the two-dimensional captured images sent from the detection unit 20 to perform human / object detection processing, and if a human is detected, further performs joint point detection processing. Furthermore, if the skeletal information generation unit 113 detects a human, it further classifies the human into a resident 71 and a non-resident (non-resident) such as staff. For example, if the skeletal information generation unit 113 detects only one person in a room, it classifies that person as a resident 71. Furthermore, if the skeletal information generation unit 113 detects multiple people in the room at the same time, it performs the following classification. In the consecutive captured images before and after that, the skeletal information generation unit 113 determines that one person who is or was present in bed 81 (lying down or sitting) is a resident 71, and classifies the other people as non-residents. In the following description, it is assumed that a person classified as a non-resident is considered to be a staff member.

[0051] The skeletal information generation unit 113 generates skeletal information using a trained model for detecting the joint points of a person from a rectangle containing a person (hereinafter also referred to as a "person rectangle"). The skeletal information includes information on the position of each joint point and skeleton of the person. The person rectangle is an area in an image that contains the joint points (nodes) of the person. Known examples of such trained models include Open Pose (https: / / arxiv.org / abs / 1812.08008) and Deep Pose (https: / / arxiv.org / abs / 1312.4659).

[0052] (Image generation unit 114) The image generation unit 114 selects an avatar model to be applied to the person detected by the skeleton information generation unit 113. If the detected person is a resident, the image generation unit 114 selects the avatar model based on the resident information. Specifically, the image generation unit 114 selects an avatar model whose appearance reflects the resident information. A plurality of avatar models are stored in the storage unit 12.

[0053] (Avatar model selection) FIG. 5 is a schematic diagram illustrating avatar models. The image generation unit 114 applies different avatar models to residents and non-residents (hereinafter referred to as staff). A common avatar model shown in FIG. 5(a) is applied to staff. For residents, the image generation unit 114 applies an avatar model in which the derivation aspect has been changed based on the resident information from the basic resident avatar model shown in FIG. 5(b). The derivation aspect includes changes in shape and / or color (including density) or texture.

[0054] (Avatar model based on resident information) In the example shown in the legend of Figure 5, if the resident information indicates a functional disorder (non-traumatic), a model (display) with a darker color (e.g., gray) is applied to the area of the functional disorder. If the resident information indicates a traumatic injury, a model with a different darker color (e.g., red or a diagonal line pattern) is applied to the area of the functional disorder. If the resident information indicates dementia, a model with a different pattern (e.g., pointillism of different sizes) is applied to the whole body or head. In addition, different basic resident avatar models may be prepared depending on the combination of gender, age, height category, and weight category described in the resident information. Here, "different" means that the avatar model's color, pattern, density, or size (such as the ratio of arm or leg thickness to the whole body) is different.

[0055] Fig. 6 is a schematic diagram showing an example of an avatar model derived according to resident information. The avatar models shown in Figs. 5 and 6 are three-dimensional avatar models corresponding to skeletal information. For example, the avatar model can be deformed according to the joint points of the skeletal information (joint information for the head, neck, wrists, elbows, shoulders, waist, knees, ankles, etc.).

[0056] Figure 6(a) shows an avatar model that is applied when the resident information does not contain any description that affects appearance or any description of functional impairment. Figure 6(a) is the same as the basic resident avatar model shown in Figure 5(b). Figure 6(b) shows an example of an avatar model that is applied when a description of pain in the left knee is extracted from the resident information. For the functionally impaired left knee, an avatar model with a spherical mark around the left knee is applied to indicate this. Similarly, Figures 6(c) to 6(e) select an avatar model according to the location of non-traumatic functional impairment or traumatic injury (functional impairment) obtained from the resident information. In Figure 6(d), an oval spherical mark (shaped like a rugby ball) is around the left elbow. Figure 6(f) shows an example of an avatar model that is applied when the resident information contains a description of mild dementia, a brain-related functional impairment.

[0057] (Image generation) Furthermore, image generation unit 114 applies the skeletal information detected by skeletal information generation unit 113 to the selected three-dimensional avatar model to generate an avatar image that reproduces the posture and movement of the person in the captured image. Then, image generation unit 114 generates an output image by superimposing the actual image of the original captured image (including a background image and a person image) and the avatar image converted to two dimensions and positioned so as to overlap (overwrite) the position of the person. The skeletal information includes joint information related to the positions of multiple joint points of the person and the connections between the joint points. Image generation unit 114 generates an avatar image using the selected avatar model from the joint information.

[0058] Furthermore, image generation unit 114 indexes the walking speed, unsteadiness during walking, and joint bending angle from the joint information, and reflects them in the avatar image. Note that when using camera 23 with a fixed angle of view instead of a real-time image (background image) of a real-time image, an image of an empty room at a certain point in time may be captured in advance, and a still image or processed image of this image may be used as the background, and an avatar image of this embodiment that reflects the skeletal information and resident information detected from the real-time captured image may be synthesized against this image.

[0059] (output unit 115) The output unit 115 transmits the output image generated by the image generation unit 114 to the terminal device 30 that has made the request, and causes it to be displayed.

[0060] (Settings reception unit 116) The setting reception unit 116 receives display settings from users such as staff members. Specifically, the setting reception unit 116 receives settings for displaying / hiding avatars and enabling / disabling the display of additional information. The contents of these settings will be described later (see FIG. 13, etc.).

[0061] (Monitoring method using the care system 1000) 7 to 12, a monitoring method performed by the information processing device 10 of the nursing care system 1000 will be described below. FIG.

[0062] (Step S01) The information processing device 10 starts executing the process of FIG. 7 in response to a request from a staff member or other watcher to display the indoor situation of the resident 71. The second acquisition unit 112 of the information processing device 10 acquires resident information of the resident. As described above, the resident information includes at least one of ADL information, level of care required, dementia level, and presence or absence of disease.

[0063] (Step S02) The first acquisition unit 111 of the information processing device 10 acquires a captured image obtained by capturing an image with the camera 23 installed in the room of the resident 71 who is the monitoring target.

[0064] (Step S03) Control unit 11 of information-processing device 10 generates an avatar image. Fig. 8 is a subroutine flowchart showing the process of step S03 in Fig. 7.

[0065] (Step S31) The skeleton information generating unit 113 recognizes objects from the captured image through the above-mentioned process, and generates skeleton information when a person is detected. Fig. 9 is a schematic diagram showing a state in which a person and a bed object are detected from the captured image and skeleton information for the person is generated.

[0066] (Step S32) If the number of people detected in step S31 is one, the image generating unit 114 proceeds to step S34, and if the number of people detected is two or more, the image generating unit 114 proceeds to step S33.

[0067] (Step S33) The image generation unit 114 regards people other than residents as staff and applies a predetermined avatar model to them (see FIG. 5(a)). Distinguishing between residents and non-residents is performed by the process described above.

[0068] (Step S34) The image generation unit 114 determines an avatar model for the resident 71. Specifically, the image generation unit 114 extracts descriptions related to functional impairment from the resident information and determines an avatar model to be applied based on the extracted information. For example, if information about pain in the left knee is extracted from the resident information, the image generation unit 114 applies an avatar model such as that shown in FIG. 6(b). Furthermore, if information about a care level of 2 and mild dementia, and information about pain in the left knee are extracted from the resident information, the image generation unit 114 applies an avatar model that combines the spherical mark that surrounds the entire circumference of the knee in FIG. 6(b) with the resident information in FIG. 6(f) (FIG. 11, described below).

[0069] (Step S35) The image generation unit 114 generates a display image. Specifically, the image generation unit 114 generates an avatar from the determined avatar model for the detected person (or each person) and skeletal information, and generates a composite image by superimposing the avatar on the captured image. The image generation unit 114 may also add annotation information. For example, this is annotation information indicating a resident or staff member. This ends the processing of FIG. 8, and the processing returns to the processing of FIG. 7.

[0070] (Step S04) The output unit 115 outputs the composite image generated in step S35. For example, the output unit 115 transmits the composite image of the terminal device 30 and causes the display unit 34 to display it.

[0071] 10 to 12 are examples of various composite images displayed in step S04. Display screen s1 in FIG. 10 displays an avatar image to which the avatar model in FIG. 6(b) is applied. The staff member watching over the patient can easily understand that the patient has pain in the right knee. Display screen s2 in FIG. 11 displays an avatar image to which the avatar model that is a combination of FIG. 5(f) and FIG. 6(b) is applied. The staff member watching over the patient can easily understand that the patient has mild dementia and pain in the right knee. Display screen s3 in FIG. 12 displays that there are two people in the room: a staff member and a resident. The staff member watching over the patient can easily understand that other staff members are providing care.

[0072] As described above, the nursing care system according to the first embodiment includes a first acquisition unit that acquires skeletal information of a person based on an image of the resident, a second acquisition unit that acquires resident information indicating the resident's condition, an image generation unit that generates an avatar image of the resident based on the resident information and skeletal information, and an output unit that displays the avatar image. This allows the resident's condition to be properly understood while protecting their privacy.

[0073] (Variation) Next, a modified example will be described with reference to Figs. 13 and 14. In this modified example, stored (recorded) captured images are used (hereinafter referred to as playback mode). Figs. 7 to 12 above show an example in which a composite image in which an avatar is superimposed on a captured image output from camera 23 is displayed in real time processing (although there is a delay of about several hundred milliseconds for image processing). The example shown below is a monitoring process that is executed when a captured image recorded in storage unit 12 is played back for later confirmation.

[0074] Fig. 13 is a subroutine flowchart showing the processing of step S03 in Fig. 7 in playback mode in a modified example. Fig. 14 shows examples of various display images according to settings. In the modified example, the processing and configuration other than those shown in Fig. 13 are the same as those in the embodiment shown in Figs. 1 to 7, etc., and therefore description thereof will be omitted.

[0075] (Step S41) The control unit 11 determines whether the operation mode is the real-time mode or the playback mode. If it is the playback mode, the control unit 11 proceeds to step S42, and if it is the real-time mode, the control unit 11 executes the processes from step S31 onwards related to the real-time processing shown in FIG.

[0076] (Step S42) The control unit 11 checks whether the avatar display is enabled. This enable / disable setting is made by the user together with an instruction to start playback, and is set, for example, through an operation screen (not shown) on the display unit 34 of the terminal device 30. If the setting indicates that the avatar display is disabled (not displayed), the control unit 11 proceeds to step S43, and if the setting is enabled, the control unit 11 proceeds to step S44.

[0077] (Step S43) Since avatar display is disabled, control unit 11 uses the captured image stored in storage unit 12 as is. In this case, in step S04, the captured image is displayed as is. FIG. 14(a) is an example of a captured image being displayed as is. On display screen s4 shown in FIG. 14(a), a recorded captured image is displayed without generating an avatar image. In the example of display screen s4, skeletal information and annotations indicating the type of recognized object are displayed in an overlapping manner, but this skeletal information, etc. may be omitted and the captured image may be displayed as is.

[0078] (Steps S44 and S45) The control unit 11 recognizes objects from the captured image, generates human skeletal information, and determines an avatar model for the resident. The processing here is the same as the processing in steps S31 and S34 in Fig. 8, and therefore a description thereof will be omitted. Note that in the example of Fig. 13, the processing in steps S32 and S33 is omitted because it is assumed that only the resident is present in the room in the captured image, but if multiple people are present in the room, the same processing may be added to the modified example of Fig. 13.

[0079] (Step S46) The control unit 11 checks whether the additional information display is enabled. This enable / disable setting is performed by the user together with an instruction to start playback, similar to the setting of the avatar display enable / disable. If the additional information display setting is disabled (hidden), the control unit 11 proceeds to step S47, and if it is enabled, the control unit 11 proceeds to step S48.

[0080] (Step S47) Since avatar display is enabled and additional information display is disabled, the control unit 11 generates an avatar and generates a composite image by superimposing the avatar on the captured image. The processing here is the same as that in step S35. FIG. 14(b) shows an example of the composite image being displayed. The display screen s5 shown in FIG. 14(b) is the same as the display screen s1 in FIG. 10, although it is not a real-time display.

[0081] (Step S48) Since the avatar display and additional information display are enabled, the control unit 11 generates an avatar and generates a composite image by superimposing the avatar on the captured image. The control unit 11 also adds additional information. The processing here is the same as that in step S35 except for the addition of the additional information. Figure 14(c) is an example of the composite image generated in step S48 being displayed and output. Additional information b0 is displayed on the display screen s6 shown in Figure 14(c). The additional information b0 is resident information. For example, the additional information is resident information used to generate the avatar, such as "Care level 1, no dementia, knee pain" or "Care level 2, dementia."

[0082] Furthermore, as shown on the display screen s6 in Fig. 14(c), an action indicator b1 may be added to the composite image. The action indicator information is information calculated from changes (movements) in skeletal information in successive frame images. For example, there are the following examples, and some of (1) to (5) may be displayed in combination. (1) Walking speed: 0.5 m / s (normally 1.0 m / s). (2) Level of unsteadiness when walking: 15 (normally level 5). (3) Knee flexion angle: 80 degrees (normally 90 degrees). (4) Time taken to stand up: 7 seconds (normally 3 seconds). (5) Transfer assistance time: 15.

[0083] In this modified example, the nursing system includes a storage unit that stores images of the resident. The nursing system also includes a first acquisition unit that acquires skeletal information of the person based on the image, a second acquisition unit that acquires resident information indicating the resident's condition, an image generation unit that generates an avatar image of the resident based on the resident information and skeletal information, and an output unit that displays the avatar image. This provides the same effect as the first embodiment, even when displaying the stored image, by protecting the privacy of the resident being monitored and appropriately grasping the resident's condition. The nursing system according to the modified example also includes a reception unit that receives settings for displaying or hiding the avatar and for displaying additional information. This allows the resident's condition to be more appropriately grasped.

[0084] The above-described configurations of the nursing care system 10000 and the information processing device 10 are the main configurations described in order to explain the features of the above embodiment, but are not limited to the above configurations and may be variously modified within the scope of the claims. Furthermore, configurations provided in general information processing devices are not excluded.

[0085] For example, the first embodiment and the modified example may be applied in combination. For example, the display of the additional information b0 and the action indicator b1 in Fig. 14(a) may be applied to the first embodiment.

[0086] Furthermore, the means and methods for performing various processes in the nursing care system or information processing device according to the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a device such as a detection unit. [Explanation of symbols]

[0087] 1000 Care System 10. Information processing equipment 11 Control section 111 First acquisition part 112 Second acquisition part 113 Skeleton Information Generation Unit 114 Image Generation Unit 115 Output section 116 Settings Reception Section 20 Detector 21 Control section 22 Communications Department 23 Camera 24 sensors 25 Care Call Department 30 Terminal Equipment 31 Control Unit 32 Storage section 33 Communications Department 34 Display section 35 Operation input section

Claims

1. a first acquisition unit that acquires human skeleton information based on an image of the resident; a second acquisition unit that acquires resident information indicating the status of the resident; an image generation unit that generates an avatar image of the resident based on the resident information and the skeleton information; an output unit that displays the avatar image; A care system equipped with:

2. The resident information includes at least one of ADL information, a level of care required, a dementia level, and the presence or absence of a disease, The nursing care system according to claim 1 , wherein the image generator generates the avatar image reflecting the resident information.

3. The nursing care system according to claim 1 , wherein the image generator generates an image in which the avatar image is superimposed on a real image captured inside a room.

4. an imaging unit that photographs the resident; a skeletal information generating unit that generates skeletal information from the two-dimensional image obtained from the imaging unit; The care system of claim 1 , comprising:

5. the skeletal information includes positions of a plurality of joint points of the person and joint information relating to connections between the joint points; The nursing care system according to claim 1 , wherein the image generator generates the avatar image from the joint information.

6. The nursing care system according to claim 5 , wherein the image generator generates indices of walking speed, unsteadiness during walking, and bending angles of joints from the joint information, and reflects the indices in the avatar image.

7. the resident information includes a care record documenting care related to the resident; The care system according to claim 1 , wherein the image generator reflects information in the care record in the avatar image.

8. If it is detected from the image that there are multiple people, The nursing care system according to claim 1 , wherein the image generating unit generates an avatar for each of a plurality of people, and generates an image to which an annotation indicating a resident or a non-resident is added.

9. a storage unit that stores the image, When displaying the image stored in the storage unit, The nursing care system according to claim 3 , wherein the image generator generates an avatar image of the resident.

10. a reception unit that receives a setting by a user as to whether to display or hide an avatar; The nursing care system of claim 9, wherein when the reception unit receives a setting to hide the image, when the image stored in the memory unit is displayed, the image generation unit displays the image without generating an avatar image of the resident.

11. a reception unit that receives display settings from a user; The nursing care system according to claim 1 , wherein the image generating unit generates a composite image using resident information of the resident as additional information in accordance with a display setting.

12. (a) acquiring skeletal information of a person based on an image of the resident; (b) acquiring resident information indicating the status of the resident; (c) generating an avatar image of the resident based on the resident information and the skeletal information; (d) displaying the avatar image; Monitoring methods, including:

13. The resident information includes at least one of ADL information, a level of care required, a dementia level, and the presence or absence of a disease, The monitoring method according to claim 12 , wherein in the step (c), the avatar image is generated to reflect the resident information.

14. (a) acquiring skeletal information of a person based on an image of the resident; (b) acquiring resident information indicating the status of the resident; (c) generating an avatar image of the resident based on the resident information and the skeletal information; (d) displaying the avatar image; A control program for causing a computer to execute a process including the above.

15. The resident information includes at least one of ADL information, a level of care required, a dementia level, and the presence or absence of a disease, The control program according to claim 14 , wherein in said step (c), the avatar image is generated to reflect the resident information.

Images