Care system, and control program

Abstract

To automatically identify a region of a bed without requiring operation by a user.SOLUTION: A care system 1000 includes: an acquisition part 111 for acquiring image data obtained by imaging an imaging region in a room where a care receiver occupies; a height information extraction part 112 for extracting height information of the imaging region; an estimation part 113 for estimating the position of a bed, from the image data obtained by imaging the bed; and a bed region identification part 114 for identifying the region of the bed, on the basis of the position of the bed estimated by the estimation part 113, and the height information extracted by the height information extraction part 112.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a care system and a control program. [Background technology]

[0002] In nursing care facilities such as hospitals and elderly care facilities, people requiring care, such as those in need of care, are at risk of falling while walking, falling out of bed and getting injured, or wandering around and going missing. Therefore, development is underway to develop a system for monitoring the condition of people being watched over so that nurses and caregivers can rush to their aid when such situations occur.

[0003] The technology disclosed in Patent Document 1 extracts the position of the bed from a distance image, and determines the person's movements as lying down, sitting up, or standing based on the positional relationship between the bed position and the detected person's area.

[0004] The information processing device (monitoring server) in Patent Document 1 includes a storage means for storing bed information indicating a preset bed size, a distance image acquisition means for acquiring a distance image of a predetermined location including the bed, and a selection receiving means for receiving a selection of a specific point on the bed on the distance image.The device then detects a first edge portion, which is a straight-line portion that forms the outline of the bed located around the specific point selected by the selection receiving means, and extracts bed information based on the length of the detected first edge portion.The device also includes an area specifying means for specifying the area of ​​the bed on the distance image based on the bed information and the endpoint coordinates of the first edge portion. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-144996 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology of Patent Document 1, the selection receiving means of the information processing device identifies the area of ​​a bed, such as a bed, by receiving the position of a specific point indicating the area of ​​the bed through a user's operation on a mobile device. Therefore, the user is forced to input the specific point to identify the area of ​​the bed for each room. Furthermore, since the location of a bed may be changed irregularly, once the location of the bed is identified, it is not necessarily possible to continue using that location. In such cases, the user is forced to operate the mobile device each time they identify a bed.

[0007] The present invention has been made in view of the above circumstances, and has an object to automatically identify the area of ​​the bed on the system side without requiring any operation by the user. [Means for solving the problem]

[0008] The above object of the present invention can be achieved by the following means.

[0009] (1) an acquisition unit that acquires image data obtained by photographing an area in a room where a care recipient resides; a height information extraction unit that extracts height information of the imaging area; an estimation unit that estimates the position of the bed from the image data of the bed; a bed area specifying unit that specifies an area of ​​the bed based on the position of the bed estimated by the estimation unit and the height information extracted by the height information extraction unit; A care system that includes:

[0010] (2) the acquisition unit acquires two pieces of image data obtained by a stereo camera arranged above the shooting area; 2. The nursing care system according to claim 1, wherein the height information extraction unit extracts height information from a parallax between the two image data.

[0011] (3) The nursing care system according to claim 1 or claim 2, wherein the estimation unit estimates the position of the bed using a trained model.

[0012] (4) The nursing care system according to claim 3, wherein the trained model is trained using a combination of an image of a room and the coordinates of a correct bed position as training data.

[0013] (5) The nursing care system according to claim 1 or claim 2, wherein the bed area identification unit extracts the outline of the bed from a point where the height information is equal to or greater than a first threshold and equal to or less than a second threshold at the position of the bed estimated by the estimation unit, and identifies the area of ​​the bed from the outline.

[0014] (6) The bed area specifying unit classifies the height information into a plurality of stages according to height, and The nursing care system of claim 1, wherein the estimation unit extracts a section with the highest occupancy rate, which is the area of ​​the points included in each section relative to the size of the rectangular area of ​​the bed estimated by the estimation unit, extracts the outline of the bed from height information of the extracted section, and identifies the area of ​​the bed from the outline.

[0015] (7) The bed area specifying unit classifies the height information into a plurality of stages, and The nursing care system according to claim 1, wherein the estimation unit extracts an outline of the bed from an area where the aspect ratio of the area of ​​the points included in each of the divisions is within a certain range in the area of ​​the bed estimated by the estimation unit, and identifies the area of ​​the bed from the outline.

[0016] (8) The nursing care system according to claim 6 or claim 7, wherein the bed area identification unit determines whether the area of ​​the points included in the extracted division is equal to or greater than a predetermined value, and if it is equal to or greater than the predetermined value, identifies the bed area.

[0017] (9) a step (a) of acquiring image data obtained by photographing an area in a room where the care recipient resides; (b) extracting height information of the imaging area; (c) estimating the position of the bed from the image data of the bed; (d) identifying a bed area based on the bed position estimated in (c) and the height information extracted in (b); A control program for causing a computer to execute a process including the above.

[0018] (10) In the step (a), two pieces of image data are acquired by a stereo camera arranged above the photographing area; 10. The control program according to claim 9, wherein in the step (b), height information is extracted from a parallax between the two pieces of image data. [Effects of the Invention]

[0019] The nursing care system of the present invention includes an acquisition unit that acquires image data obtained by photographing an area within a room where a care recipient resides, a height information extraction unit that extracts height information of the photographed area, an estimation unit that estimates the position of the bed from image data of the bed, and a bed area identification unit that identifies the area of ​​the bed based on the position of the bed estimated by the estimation unit and the height information extracted by the height information extraction unit. This allows the area of ​​the bed to be identified automatically without requiring user operation. [Brief explanation of the drawings]

[0020] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for purposes of illustration only and are not intended to be limiting. [Figure 1] 1 is a schematic diagram showing a care system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a detection unit. [Figure 3] FIG. 2 is a block diagram showing the configuration of a terminal device. [Figure 4]FIG. 1 is a block diagram showing a configuration of an information processing device. [Figure 5] FIG. 10 is a schematic diagram showing the data flow of the entire bed region identification process. [Figure 6] These are examples of two image data obtained from a stereo camera. [Figure 7] FIG. 10 is a diagram showing an object and its position information estimated by an estimation unit. [Figure 8] FIG. 10 is a diagram showing height information extracted by a height information extraction unit. [Figure 9] 10 is a flowchart showing a method for identifying a bed area. [Figure 10] 10 is a subroutine flowchart showing the bed region specifying process in step S05 of FIG. 9. [Figure 11] FIG. 10 is a schematic diagram showing a process for identifying the coordinates of a bed area. [Figure 12] 10 is a subroutine flowchart showing a bed region specifying process in step S05 in the second embodiment. [Figure 13] FIG. 2 is a schematic diagram showing an example of a rectangular bed region and the area of ​​each section. [Figure 14] 13 is a subroutine flowchart showing a bed region specifying process in step S05 in the third embodiment. [Figure 15] 10A and 10B are schematic diagrams showing examples of aspect ratios in cluster regions in each section. [Figure 16] 10 is a flowchart showing a method for specifying a bed area in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0022] FIG. 1 is a schematic diagram showing a nursing care system 1000 according to a first embodiment. The nursing care system 1000 includes an information processing device 10, a detection unit 20, and a terminal device 30, which are connected to each other via a network 50 for communication. The information processing device 10 is a server or a personal computer (PC). When the information processing device 10 is configured as a server, it may be an on-premise server installed in a nursing care facility or a cloud server using a commercial cloud service. The terminal device 30 is a PC and is operated by users such as nursing care staff (hereinafter simply referred to as staff) or administrators at the nursing care facility. The terminal device 30 may also be a mobile device such as a smartphone used by the staff. During work hours, the staff use the terminal device 30 to perform their work. As will be described in detail later, the nursing care system 1000 generates bed area data by identifying bed areas. The care system 1000 uses the bed area data through the detection unit 20 to monitor the state of the care recipient (hereinafter referred to as the resident 71) in the room, such as getting up, falling, tipping over, sleeping, etc. The monitoring results are recorded in the information processing device 10. Furthermore, when some specific state such as getting up, falling, tipping over, etc. is detected, an event is notified to the terminal device 30 carried by the staff on duty, and the staff takes appropriate measures.

[0023] Hereinafter, the detection unit 20 will be described first, followed by the terminal device 30 and the information processing device 10. In addition, in the nursing care system 1000 shown in FIG. 1, a detection unit 20 for each of multiple rooms and a terminal device 30 are arranged in one nursing care facility. An example is shown in which each device in the nursing care facility is connected to the information processing device 10 via a network 50. That is, each device 20, 30 in one nursing care facility is connected to the information processing device 10. However, this is not limited to this, and one information processing device 10 may be connected to multiple nursing care facilities, each having a terminal device 30 and multiple detection units 20.

[0024] (Detection unit 20) FIG. 2 is a block diagram showing the configuration of the detection unit 20. Referring to FIGS. 1 and 2, the detection unit 20 is configured to monitor the movements (behavior) of the resident 71 in the observation area (room) of a room where the resident 71 residing in the nursing care facility is located, with the observation area being the interior of the room. The resident 71 is a person receiving care and residing in the nursing care facility. Care or nursing care is provided to the resident 71 by staff. The room is, for example, one room in a nursing care facility where multiple residents 71 reside. Furthermore, the information processing device 10 is connected to the detection unit 20 provided in each of the multiple rooms as shown in FIG. 1. The device ID of each detection unit 20 and the subject ID of the resident 71 to be monitored are associated and stored in a memory unit (memory unit 32 or memory unit 12 described below).

[0025] As shown in FIG. 2, the detection unit 20 includes a control unit 21, a communication unit 22, an imaging unit 23, a sensor 24, a care call unit 25, etc. The detection unit 20 generates sensing data (image data, audio data, etc.) by constantly sensing the resident 71 in the observation area (also referred to as the photographing area) in real time, 24 hours a day, using a plurality of various sensors. The control unit 21 may also include a large-capacity memory. The imaging unit 23 and the sensor 24 are disposed in the main body of the detection unit 20. The main body is disposed on the ceiling or upper part of the wall of the room. The care call unit 25 is disposed separately from the main body and is communicatively connected to the main body via wired or short-range wireless communication.

[0026] The control unit 21 is composed of a CPU, RAM, ROM, etc., and controls each part of the detection unit 20 and performs calculation processing according to a program. The communication unit 22 is an interface circuit (for example, a LAN card, etc.) for communicating with other devices such as the terminal device 30 via the network 50.

[0027] The imaging unit 23 captures an image of the room in which the bed 81 in which the resident is located as an imaging area, and outputs a two-dimensional image (captured image). For example, it is disposed on the ceiling or upper part of the wall of the room, and captures an overhead image of the room in which the resident 71 is located directly below as an observation area (imaging area). The imaging unit 23 is a near-infrared camera or a visible light camera. The imaging unit 23 is, for example, a stereo camera having a first camera 231 and a second camera 232. The first camera 231 and the second camera 232 are disposed so that their optical axes extend in a substantially vertical direction, providing an overhead view of the imaging area of ​​the room including the bed 81. The optical axes of the first camera 231 and the second camera 232 are parallel to each other. The two optical axes are disposed a predetermined distance apart (hereinafter also referred to as the optical axis distance) in a direction perpendicular to the optical axes. The optical axis distance is several tens of centimeters, for example, 20 cm. A distance image is generated from two sets of image data (hereinafter referred to as image data 1 and 2) captured simultaneously by the first and second cameras 231 and 232, with each pixel storing the distance value from the subject to the camera, calculated based on the parallax information. Because this distance image is obtained from image data from a stereo camera with a bird's-eye view, it is also referred to as "height information" below. The process of generating the distance image will be described later. The conversion from the distance image to height information is performed using the following method. While the distance value is defined as the distance from the camera, the height information is the distance from a reference surface (floor). Therefore, the conversion to height information is performed by subtracting the distance value of each pixel from the distance to the floor, which is set for each position (direction) of the pixel (point) in the distance image.

[0028] The focal length, pixel size, optical axis direction, distortion, and optical axis distance of the optical system of the imaging unit 23 are registered in advance and stored in the storage unit 12 or the storage area of ​​the control unit 21. Furthermore, by performing calibration, the real space in the room is associated with the camera coordinate system. For example, calibration is performed by the user inputting the distance (height) and direction to multiple markers captured on the fixed imaging unit 23. Furthermore, the distance from the floor for each pixel (each direction) is registered by calibration.

[0029] The sensor 24 detects the movement of the resident 71. The sensor 24 includes various sensors other than the imaging unit 23. For example, the sensor 24 may include at least one of a body movement sensor, a bed sensor, a mat sensor, a thermal sensor, and an infrared sensor. The "body movement sensor" may be a Doppler shift sensor that transmits and receives microwaves to and from the bed 81 and detects the Doppler shift of the microwaves caused by the body movement (e.g., breathing) of the resident 71. This body movement sensor detects chest movement (up and down movement of the chest) associated with the breathing of the resident 71. The period and amplitude of the body movement can be used to determine the sleep state or abnormal micro-movements (e.g., due to cardiac arrest). The "bed sensor" may be a sensor attached to the bed. For example, the bed sensor may detect weight and be placed on the bed 81 or on the floor at the exit of the bed 81 as an observation area to detect whether a person is standing on the sensor or to detect the sleep state. The "mat sensor" has the same function as the bed sensor, detecting the presence of a person in each divided area of ​​the floor. The "infrared sensor," also known as a human presence sensor, detects whether a person is present in the room. For example, the infrared sensor is placed throughout the room or on the bed 81 as its observation area.

[0030] The care call unit 25 includes a push button switch, and detects a care call when the switch is operated by the resident 71.

[0031] (Terminal device 30) FIG. 3 is a block diagram showing the configuration of the terminal device 30. As shown in FIG. 3, the terminal device 30 includes a control unit 31, a memory unit 32, a communication unit 33, a display unit 34, and an operation input unit 35. The control unit 31 is composed of a CPU, RAM, ROM, etc., and controls each unit of the terminal device 30 and performs calculations according to a program. The memory unit 32 is composed of a hard disk or the like that stores various programs and data. The communication unit 33 is an interface circuit for communicating with other devices via a network 50, either wired or wirelessly. The display unit 34 is composed of an LCD display, a touch sensor, etc., and displays various information and an operation screen. The operation input unit 35 is an input device such as a keyboard, a mouse, a touch sensor, etc., and receives input from staff, staff leaders of units (also referred to as areas or groups in charge), and facility managers (hereinafter collectively referred to as managers, etc.).

[0032] (Information processing device 10) The configuration of the information processing device 10 will be described below with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device.

[0033] 4, the information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is composed of a CPU, RAM, ROM, etc., and controls each unit of the information processing device 10 and performs calculation processing according to a program.

[0034] (Control unit 11) The control unit 11 functions as an acquisition unit 111 in cooperation with the communication unit 13. The control unit 11 also functions as an estimation unit 112, a height information extraction unit 113, a bed area identification unit 114, an event determination unit 115, and an event notification unit 116. Each function of the control unit 11 will be described later. The communication unit 13 is an interface circuit for communicating with other devices via the network 50.

[0035] (Storage unit 12) The storage unit 12 is composed of a hard disk etc. that stores various programs and various data. The storage unit 12 stores trained models, bed-related data, resident information, staff information, event lists etc.

[0036] The "trained model" is used for person and object detection processing from image data (particularly image data 1). In the person and object detection processing, this trained model detects areas in the image data where objects, including people, exist as object presence areas. Then, it calculates a confidence score for each predetermined category of objects contained in the detected object presence area. The confidence score is the likelihood of the target object. The person and object detection processing may calculate the confidence score using a known technique using a deep neural network (DNN). The predetermined categories may be, for example, people, chairs, furniture, and beds (bedding). In the person and object detection processing, the object presence area with the highest confidence score in the person category is detected as a person rectangle (bounding box). In other words, in the person and object detection processing, the resident 71 is detected as an object (moving body). Similarly, the object presence area with the highest confidence score in a predetermined object category is detected as the object rectangle (e.g., the object area of ​​a bed) of the category with the highest confidence score.

[0037] This trained model may be adjusted to improve the accuracy of bed detection. For example, a trained model previously trained on a large dataset may be applied to the task of bed detection. This adjustment (also known as transfer learning or fine-tuning) is performed by retraining the trained model using training data that combines images of the room and ground truth data indicating the coordinates of the correct bed position (object rectangle).

[0038] The "bed-related data" includes (1) first and second thresholds that are the range of bed height information, (2) bed size, and (3) bed area data. The first and second thresholds in (1) are, for example, 50 cm and 70 cm, respectively. Note that these first and second threshold values ​​are merely examples. The first threshold may be set to any value in the range of 30 to 60 cm, and the second threshold may be set to any value in the range of 60 to 90 cm that is greater than the first threshold. (2) is, for example, 180 cm long and 90 cm wide (size on the XY plane). (1) and (2) are default values ​​that are commonly applied to each room in the facility, but may be set for each room by operating the screen via the terminal device 30. (3) Bed area data is information indicating the position of the bed 81 for each room identified by the bed area identification process (described later in FIGS. 9 and 10).

[0039] "Resident information" is information about the resident 71, and includes the room number in which the resident 71 resides, family information, the age, sex, medical history, and medical conditions of the resident 71.

[0040] "Staff information" includes information such as staff name, affiliated unit, and work schedule.

[0041] The event list includes information on the type of detected event, the time of occurrence, the person who caused the event (room number), and the response status for the event.

[0042] (Control unit 11) Hereinafter, each function of the control unit 11 will be described with reference to Figures 5 to 8 as well as Figure 4. Figure 5 is a schematic diagram showing the data flow of the entire bed region specification process.

[0043] (Acquisition part 111) The acquisition unit 111 acquires, from the detection unit 20, image data obtained by imaging with the imaging unit 23, which is arranged to capture images of the inside of a room where a resident is present. More specifically, image data 1 is acquired from the first camera 231 of the imaging unit 23, and image data 2 is acquired from the second camera 232. Fig. 6 shows an example of image data 1 and 2 acquired from the first and second cameras 231 and 232 on the left and right, respectively.

[0044] (Estimation part 112) The estimation unit 112 uses the trained model to perform human / object detection processing from one of the acquired image data 1 and 2 (e.g., image data 1). FIG. 7 is a diagram showing object position information detected by the human / object detection processing of the estimation unit 112. In the example shown in FIG. 7, the object region (rectangle) of the bed and the object region (rectangle) of the chair are shown together with their reliability scores. The object region of the bed is assigned information indicating the range of the rectangle (bounding box), for example, the XY coordinates of the four corners. In the following description, the XY coordinates or XYZ coordinates (also referred to as XYH coordinates, where H is the height from the floor) are described as being in a real-space coordinate system converted from the coordinate system of the first camera, etc.

[0045] (Height information extraction unit 113) The height information extraction unit 113 extracts height information of the captured area. Specifically, the height information extraction unit 113 performs distance measurement to measure the distance from the camera based on the parallax between two pieces of image data 1 and 2 acquired from the stereo camera (first and second cameras 231 and 232), generating a distance image. The height information extraction unit 113 then refers to pre-stored distance values ​​to the floor for each pixel, converts this distance image into height information, and outputs it. For example, to measure the distance to the floor, the nursing care system 1000 performs distance measurement using the two pieces of image data 1 and 2 in a state where there are no people and as few objects as possible so that the floor is visible, in response to a user's instruction. The information processing device 10 of the nursing care system 1000 then assumes that the room is a rectangular parallelepiped or a three-dimensional object formed by combining rectangular parallelepipeds, i.e., that the floor is flat and perpendicular to the side walls. It estimates and stores the distances from the visible floor and side wall surfaces to the entire range of the floor in the X and Y directions, as well as the distance to the floor in each pixel direction of the distance image.

[0046] Fig. 8 is an example of height information extracted by the height information extraction unit 113. In the example of Fig. 8, the height is converted into a density value and displayed within a range of 0 to 256 cm from the floor surface.

[0047] While the examples shown in FIGS. 2 and 5 illustrate examples of generating height information from images captured by a stereo camera, the present invention is not limited thereto. A distance image may be obtained using a structured light method and used to extract height information. Specifically, the detection unit 20 includes a camera and a projector positioned a distance equivalent to the optical axis distance from the camera. The projector then projects a grid-like pattern of light into the image capture area, and the device detects the distance from the camera to the object by analyzing distortions and deviations of the pattern of light according to the surface shape of the object in the image capture area. Alternatively, the detection unit 20 may include a ToF distance sensor separate from the camera. For example, the detection unit 20 may include a LiDAR (Light Detection and Ranging) sensor, which generates a distance image and obtains height information. In either method, pixel positions in the image data 1 are associated with pixel positions in the height information (distance image) through calibration.

[0048] (Bed area identification unit 114) The bed area identifying unit 114 identifies the bed area from the object position information and height information detected by the estimation unit 112. The bed area identifying unit 114 extracts height information of the range corresponding to the position information of the bed detected by the estimation unit. Then, it identifies the bed area from points (pixels) having height information of the range corresponding to the height of the bed. This bed area identifying method will be described in detail later (see FIG. 10, etc.). In this specification, the terms area and position are defined as follows: The bed area detected by the estimation unit 112 is called the "bed position (bed position)" or "bed rectangular area," and is distinguished from the bed area (bed area) identified by the bed area identifying unit 114, described later, from the height information and bed rectangular area. The bed position detected by the estimation unit 112 is composed of information on the upper left coordinate and length and width (length in the X and Y directions), or coordinate information of the four corners (X and Y coordinates).

[0049] (Event determination unit 115) The event determination unit 115 determines an event from the image data 1 obtained from the first camera. The events include "getting up" when the resident 71 gets up from the bed 81, "getting out of bed" when the resident 71 leaves the bed 81, "falling" when the resident 71 falls off the bed 81, and "falling" when the resident 71 falls onto the floor or the like. The events also include "sleeping" determined based on both the image data 1 and data from another sensor 24 (for example, a body movement sensor).

[0050] The event determination unit 115 detects the silhouette of the entire body of the resident 71 (hereinafter referred to as "human silhouette") from multiple frames of image data 1 (moving images). A head silhouette (neither of which is shown) may be used in addition to or instead of the human silhouette. The human silhouette can be detected, for example, by extracting a range of pixels with a relatively large difference using time subtraction, which subtracts images taken at different times. The human silhouette may also be detected using a background subtraction method, which subtracts a background image from the image data 1. Whether the resident 71 is getting up, getting out of bed, falling, or tripping is determined from the detected human silhouette based on the posture of the resident 71 (e.g., standing, sitting, lying down, etc.) and their position relative to objects in the room, such as the bed 81. Here, the bed area data indicating the position of the bed 81 is identified by the bed area identification unit 114 described above. For example, a determination of "getting up" can be recognized when the width of the human silhouette crossing any side of a rectangle in a top view, with vertices at the four corners of the bed 81, which has been specified in advance by the bed area specifying unit 114, increases to 20 cm or more. Also, for example, getting out of bed can be recognized when the ratio of the area of ​​the human silhouette outside the rectangle to the area inside the rectangle increases to 80% or more. The determined event is recorded in an event list together with the event type and occurrence status (time, room ID, occupant (subject ID), etc.).

[0051] (Event notification unit 116) If the type of event that has occurred is of a specific type, for example, a fall or getting out of bed, the event notification unit 116 notifies the terminal device 30 of the staff member on duty. One of the multiple staff members who receives this notification responds to the event and inputs from the terminal device 30 that the event has been responded to or will be responded to. Upon receiving the report that the event has been responded to from the terminal device 30, the control unit 11 changes the status of the event from "not responded to" to "responded to" and records it in the event list.

[0052] (Method for identifying bed area by care system 1000) 9 to 11, a method for specifying a bed area performed by the information processing device 10 of the nursing care system 1000 will be described below. Fig. 9 is a flowchart showing the method for specifying a bed area performed by the information processing device 10.

[0053] (Step S01) The information processing device 10 determines whether it is time to update the bed area. For example, when a user issues an instruction to do so via the terminal device 30, or at a predetermined regular interval, for example, once a day at a preset time. If it is time to do so, the control unit 11 advances the process to step S02. The bed area is updated for each room.

[0054] (Step S02) The acquisition unit 111 acquires image data 1 and 2 obtained by first and second cameras 231 and 232 capturing images of the interior of a room as a capture area.

[0055] (Step S03) The estimation unit 112 uses the trained model to estimate an object and its position from the image of the image data 1. The categories of objects to be estimated include chairs, beds, etc. When an object is detected, the type of object and its position are indicated (see FIG. 7). FIG. 11 is an explanatory diagram showing the states corresponding to each process. When a bed 81 is detected as shown in data d1, its type and its position (rectangular area) are detected.

[0056] (Step S04) The height information extraction unit 113 performs distance measurement processing to measure the distance from the camera based on the parallax between the two image data 1 and 2 as described above, generates a distance image, and extracts height information based on this and the distance to the floor.

[0057] (Step S05) The bed area specifying unit 114 specifies the bed area based on the information obtained up to this point. Fig. 10 is a subroutine flowchart showing the bed area specifying process in step S05 of Fig. 9.

[0058] (Step S501) The bed area specifying unit 114 extracts height information corresponding to the rectangular area of ​​the bed estimated from the image data. As shown by the arrow a1 and data d3 in Fig. 11, height information present in the area corresponding to the rectangular area of ​​the bed estimated from the image data is extracted.

[0059] (Step S502) The bed region identifying unit 114 extracts points (pixels) whose height is within a predetermined range, specifically, points (pixels) that are equal to or greater than a first threshold and equal to or less than a second threshold (data d4). For example, the first and second thresholds are 50 cm and 70 cm, respectively. Default values ​​for the first and second thresholds are stored in the storage unit 12. In data d4 of FIG. 11, the solid pixel in the center is a point whose height value is within the predetermined range.

[0060] (Step S503) The bed region specifying unit 114 specifies the outline from the points extracted in step S502. For example, the bed region specifying unit 114 extracts the outer edges of the points extracted in step S502 and detects multiple corners from the edges (data d5). The bed region specifying unit 114 then specifies the outline from the corners that are farthest from the center in four directions and specifies the four corners as the coordinates of the four corners that define the bed region (data d6). This obtains the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the four corners that indicate the bed region. This ends the processing of FIG. 10, and the processing returns to the processing of FIG. 9 (return).

[0061] (Step S06) The control unit 11 registers the bed area identified in step S05 in the storage unit 12, and then ends the process. After that, the event determination unit 115 detects various events using the information on this registered bed area.

[0062] In this way, in the nursing care system according to the first embodiment, the identification unit identifies the area of ​​the bed based on the position of the bed estimated from the image data by the estimation unit and the height information extracted by the height information extraction unit. In this way, the nursing care system can automatically identify the area of ​​the bed with high accuracy.

[0063] (Second embodiment) Next, the bed area identification process in the second embodiment will be described with reference to Fig. 12 and Fig. 13. In the bed area identification process, the nursing care system 1000 according to the second embodiment classifies height information into multiple stages according to height (height value) as shown below, and identifies the bed area using the height information after classification. In this respect, it differs from the first embodiment, but the configuration examples of the first embodiment shown in Figs. 1 to 9 and 11 can be commonly applied to the other configurations. Fig. 12 is a subroutine flowchart showing the bed area identification process in step S05 in the second embodiment.

[0064] (Step S521) The bed area specifying unit 114 extracts height information corresponding to the bed area estimated from the image data. The processing here is the same as that in step S501 described above. As indicated by the arrow a1 and data d3 in Fig. 11, height information present within the area corresponding to the rectangular area of ​​the bed estimated from the image data is extracted.

[0065] (Step S521) The bed area specifying unit 114 divides the extracted height information into section 1 to section n. Each section is divided by the same interval. For example, the bed area specifying unit 114 divides the range from the first threshold value to the second threshold value into a plurality of sections of equal intervals. Here, the bed area specifying unit 114 divides the height information into the following three sections (n=3): (Category 1) 50-56.7cm (50cm or more and less than 56.7cm. The same applies below.) (Category 2) 56.7~63.4m, (Category 3) 63.4~70cm.

[0066] (Step S523) Next, the bed region specifying unit 114 calculates the areas b1 to b3 of the sections 1 to 3. The area b may be calculated using the number of points (number of pixels) belonging to the section itself, or the xy area of ​​the imaging region according to the angle of view and the number of points may be converted into an area from the number of pixels.

[0067] (Step S524) The bed area specifying unit 114 extracts the area with the highest occupancy rate from among the sections 1 to n. The occupancy rate is the ratio of area b to area a of the rectangular area of ​​the bed (occupancy rate = b / a). Area a is the area of ​​the position of the bed (i.e., the rectangular area) estimated by the estimation unit 112, shown as data d1 in FIG. 11. FIG. 13 is a schematic diagram showing an example of the area a of the rectangular area of ​​the bed and areas b1 to b3 of each section. In the example shown in FIG. 13, area b1 of section 1 is the largest, showing the highest occupancy rate. The bed area specifying unit 114 extracts section 1, which has the highest occupancy rate.

[0068] (Step S525) Here, the bed area specifying unit 114 determines whether the area b of the extracted section is equal to or greater than a predetermined value v. The predetermined value v is set in advance and stored in the storage unit 12. For example, the predetermined value v is set based on the area of ​​16,200 cm^2 of a typical bed with a major axis length of 180 cm and a minor axis length of 90 cm. For example, the predetermined value v is set to 50% of the typical area of ​​a bed. If the area b is equal to or greater than the predetermined value v, the bed area specifying unit 114 proceeds to step S526. On the other hand, if the area b is less than the predetermined value v, the processing of FIG. 12 ends. In this case, an error determination may be performed.

[0069] (Step S526) The bed region specifying unit 114 specifies the outline from the points of the section extracted in step S524. The processing here is the same as that in step S503. For example, the bed region specifying unit 114 similarly extracts the outer edges of the points extracted from the section extracted in step S524 and detects multiple corners from the edges (data d5 in FIG. 11). Then, the bed region specifying unit 114 specifies the outline from the corners that are farthest from the center in four directions and specifies the four corners as the coordinates of the four corners that define the bed region (data d6). This obtains the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the four corners that indicate the bed region. This ends the processing in FIG. 12, and the processing returns to the processing in FIG. 9 (return).

[0070] As described above, in the nursing care system according to the second embodiment, the bed area identification unit 114 divides the height information into multiple levels according to the height, extracts the area corresponding to the bed position estimated by the estimation unit 112, and extracts the area with the highest occupancy rate, which is the area of ​​the points included in each area. The unit then extracts the outline of the bed from the height information of the extracted area, and identifies the area of ​​the bed from the outline. This achieves the same effect as the first embodiment. That is, the nursing care system can automatically identify the area of ​​the bed with high accuracy.

[0071] (Third embodiment) Next, the bed area identification process in the third embodiment will be described with reference to Fig. 14 and Fig. 15. As in the second embodiment, the nursing care system 1000 according to the third embodiment classifies height information into multiple stages according to height (height value) in the bed area identification process, and identifies the bed area using the height information after classification. The configuration examples of the first embodiment shown in Figs. 1 to 9 and 11 can also be applied to the third embodiment. Fig. 14 is a subroutine flowchart showing the bed area identification process in step S05 in the third embodiment.

[0072] (Steps S541, S542) The processing here is the same as that of steps S521 and S522 shown in Fig. 12, and therefore a description thereof will be omitted. The bed area specifying unit 114 obtains the divided height information. In the following, as in the description of the second embodiment, the bed area specifying unit 114 will be described as dividing the height information into the following three sections (n=3):

[0073] (Step S543) The bed region identifying unit 114 clusters the points in each of sections 1 to 3 and calculates the aspect ratios AR1 to AR3 of the resulting clusters (clustered regions). The aspect ratio AR is calculated using the length in the X-axis direction and the length in the Y-axis direction. The aspect ratio is calculated by dividing the minor axis length by the major axis length, with the longer side being the major axis and the shorter side being the minor axis. The directions of the X-axis and Y-axis are set in advance and are along the side wall surfaces of the room. Note that there are also cases where the bed 81 is placed at an angle rather than along the wall surfaces of the room. For this reason, the bed region identifying unit 114 may rotate the cluster by a predetermined angle (in the range of 0 to 90 degrees), calculate the aspect ratio AR at each angle, and use the largest aspect ratio AR.

[0074] (Step S544) The bed region specifying unit 114 extracts the section in which the aspect ratio AR calculated in step S543 is closest to a predetermined value w. The predetermined value w is set in advance and stored in the storage unit 12. For example, the predetermined value w is set to 0.5, which is the ratio of the length and width of a typical bed, 90 cm / 180 cm.

[0075] (Step S545) Here, the bed area specifying unit 114 calculates the area of ​​the selected section and determines whether the calculated area b is equal to or greater than a predetermined value v. The processing here is the same as that in step S525. If the calculated area b is equal to or greater than the predetermined value v, the bed area specifying unit 114 proceeds to step S546. On the other hand, if the calculated area b is less than the predetermined value v, the processing in FIG. 14 ends.

[0076] (Step S546) The bed region specifying unit 114 specifies the outline from the points of the section extracted in step S544. The processing here is the same as that in step S503. For example, the bed region specifying unit 114 similarly extracts the outer edges of the points extracted from the section extracted in step S524 and detects multiple corners from the edges (data d5 in FIG. 11). Then, the bed region specifying unit 114 specifies the outline from the corners that are farthest from the center in four directions and specifies the four corners as the coordinates of the four corners that define the bed region (data d6). This obtains the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the four corners that indicate the bed region. This ends the processing in FIG. 14, and the processing returns to the processing in FIG. 9 (return).

[0077] As described above, in the nursing care system according to the third embodiment, the bed area specifying unit 114 divides the height information into multiple stages according to the height, and extracts the outline of the bed from an area of ​​the bed area estimated by the estimation unit 112 where the aspect ratio of the area of ​​the points included in the division is within a certain range, and specifies the bed area from the outline. In this way, the same effects as those of the first and second embodiments can be obtained. That is, the nursing care system can automatically specify the bed area with high accuracy.

[0078] (Variation) In the above-described first to third embodiments, examples have been shown in which default values ​​for setting values ​​used to control bed-related data and the like are stored in advance in the storage unit 12. However, bed sizes are not limited to standard sizes, and some care recipients may use beds with special sizes, heights, or shapes. In the following modified example, these default values ​​are changed to accommodate a variety of beds. Fig. 16 is a flowchart showing a method for identifying a bed area in the modified example.

[0079] (Step S10) The control unit 11 accepts various setting changes regarding the floor size and height of the bed. These setting changes are made irregularly for each room, for example, when a care recipient first moves into the room or when the bed in the room is replaced.

[0080] The control unit 11 accepts setting inputs from a user such as a staff member via an operation screen displayed on the terminal device 30. For example, at least one of the following settings is accepted. (1) A predetermined value v relating to the area and a predetermined value w relating to the aspect ratio AR as information relating to the plane size of the bed (steps S525, S544, S545), (2) Information about the height size of the bed, including the first and second thresholds (step S501, etc.) and the interval width for classification.

[0081] The received settings are stored in the storage unit 12 as setting values ​​for each room.

[0082] (Steps S11 to S16) In the following, the control unit 11 reads out various settings set for each room from the storage unit 12 and performs the processes of steps S11 to S16. The processes of steps S11 to S16 are the same as steps S01 to S06 in the first embodiment shown in Fig. 9, and therefore, description thereof will be omitted.

[0083] In this way, in the modified example, the set value changed according to the bed in each room can be applied. Note that the process of step S10 may be performed if an error occurs in the process of step S525 (NO).

[0084] The above-described configurations of the nursing care system 1000 and the information processing device 10 are the main configurations described in order to explain the features of the above embodiment, but are not limited to the above configurations and can be variously modified within the scope of the claims. Furthermore, configurations provided in general information processing devices are not excluded.

[0085] For example, the first to third embodiments can be applied in combination. Also, some of the functional units of the control unit 11 may be performed by the detection unit 20. For example, the functions of the estimation unit 112 and the height information extraction unit 113 may be executed by the control unit 21 of the detection unit 20. In this case, the control unit 11 acquires the object detection result and height information from the detection unit 20. Also, as another example, although the example in which the nursing care system 1000 includes the information processing device 10, the detection unit 20, and the terminal device 30 has been described, the nursing care system 1000 may be configured only by the information processing device 10.

[0086] Furthermore, the means and methods for performing various processes in the nursing care system or information processing device according to the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a device such as a detection unit.

[0087] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims. [Explanation of symbols]

[0088] 1000 Care System 10. Information processing equipment 11 Control section 111 Acquisition Department 112 Estimation Department 113 Height information extraction unit 114 Bed Area Identification Unit 115 Event Judgment Unit 116 Event Notification Section 20 Detector 21 Control Unit 22 Communications Department 23 Imaging unit 231 Camera 1 232 Second Camera 24 sensors 25 Care Call Department 30 Terminal Equipment 31 Control Unit 32 Storage section 33 Communications Department 34 Display section 35 Operation input section

Claims

1. an acquisition unit that acquires image data obtained by capturing an image of a capture area in a room where the care recipient resides; a height information extraction unit that extracts height information of the imaging area; an estimation unit that estimates the position of the bed from the image data of the bed; a bed area specifying unit that specifies an area of ​​the bed based on the position of the bed estimated by the estimation unit and the height information extracted by the height information extraction unit; A care system that includes:

2. the acquisition unit acquires two pieces of image data obtained by a stereo camera arranged above a shooting area, The nursing care system according to claim 1 , wherein the height information extraction unit extracts height information from a parallax between the two image data.

3. The nursing care system according to claim 1 or 2, wherein the estimation unit estimates the position of the bed using a trained model.

4. The nursing care system according to claim 3 , wherein the trained model is trained using a combination of an image of a room and a correct coordinate of a bed position as training data.

5. 3. The nursing care system according to claim 1, wherein the bed area identification unit extracts the outline of the bed from a point at the position of the bed estimated by the estimation unit where the height information is equal to or greater than a first threshold and equal to or less than a second threshold, and identifies the area of ​​the bed from the outline.

6. The bed area specifying unit classifies the height information into a plurality of stages according to height, and The nursing care system according to claim 1, wherein the estimation unit extracts the section with the highest occupancy rate, which is the area of ​​the points included in each section relative to the size of the rectangular area of ​​the bed estimated by the estimation unit, extracts the outline of the bed from height information of the extracted section, and identifies the area of ​​the bed from the outline.

7. The bed area specifying unit classifies the height information into a plurality of stages, and The nursing care system according to claim 1, wherein the estimation unit extracts the outline of the bed from an area where the aspect ratio of the area of ​​the points included in each of the divisions is within a certain range in the area of ​​the bed estimated by the estimation unit, and identifies the area of ​​the bed from the outline.

8. The nursing care system according to claim 6 or claim 7, wherein the bed area identification unit determines whether the area of ​​the points included in the extracted division is equal to or greater than a predetermined value, and if the area is equal to or greater than the predetermined value, identifies the bed area.

9. a step (a) of acquiring image data obtained by photographing an area in a room where a care recipient resides; (b) extracting height information of the imaging area; (c) estimating the position of the bed from the image data of the bed; (d) identifying a bed area based on the bed position estimated in (c) and the height information extracted in (b); A control program for causing a computer to execute a process including the above.

10. In the step (a), two sets of image data are acquired by a stereo camera arranged above a photographing area; 10. The control program according to claim 9, wherein in said step (b), height information is extracted from a parallax between said two pieces of image data.

Images